AI-generated Article Summarizer Python, NLP

```python
import nltk
from nltk.corpus import stopwords
from nltk.tokenize import word_tokenize, sent_tokenize
from nltk.probability import FreqDist
from heapq import nlargest

# Download necessary NLTK resources (only needed the first time)
# nltk.download('punkt')
# nltk.download('stopwords')


def summarize_article(article_text, num_sentences=3):
    """
    Summarizes a given article text into a specified number of sentences.

    Args:
        article_text (str): The text of the article to summarize.
        num_sentences (int): The number of sentences to include in the summary.  Defaults to 3.

    Returns:
        str: The summary of the article.
    """

    # 1. Text Preprocessing

    # Tokenize the text into sentences
    sentences = sent_tokenize(article_text)

    # Tokenize the text into words
    words = word_tokenize(article_text)

    # Remove stop words and punctuation
    stop_words = set(stopwords.words("english"))  # Get common English stop words
    punctuation = ['.', ',', '!', '?', ';', ':', '"', "'", '(', ')', '[', ']', '{', '}', '-', '_']
    filtered_words = [word.lower() for word in words if word.isalnum() and word.lower() not in stop_words and word not in punctuation]


    # 2. Calculate Word Frequencies

    # Calculate the frequency of each word
    word_frequencies = FreqDist(filtered_words)

    # 3. Score Sentences Based on Word Frequencies

    # Create a dictionary to store sentence scores
    sentence_scores = {}

    for sentence in sentences:
        # For each sentence, count the frequency of important words
        for word in word_tokenize(sentence.lower()):  # Lowercase for consistent matching
            if word in word_frequencies:
                if len(sentence.split(' ')) < 30:  # Optionally penalize very long sentences
                    if sentence in sentence_scores:
                        sentence_scores[sentence] += word_frequencies[word]
                    else:
                        sentence_scores[sentence] = word_frequencies[word]

    # 4. Select Top Sentences for Summary

    # Get the top 'num_sentences' sentences with the highest scores
    summary_sentences = nlargest(num_sentences, sentence_scores, key=sentence_scores.get)  # Use heapq for efficiency

    # 5. Combine Sentences into Summary

    # Join the top sentences into a coherent summary
    summary = ' '.join(summary_sentences)

    return summary



# Example Usage:
if __name__ == "__main__":
    article = """
    Artificial intelligence (AI) is revolutionizing various aspects of our lives, from healthcare to transportation.  AI algorithms are being used to diagnose diseases more accurately, develop personalized treatments, and accelerate drug discovery. In the field of transportation, self-driving cars are becoming a reality, promising to reduce accidents and improve traffic flow. However, the widespread adoption of AI also raises concerns about job displacement and ethical considerations. It is crucial to develop AI responsibly, ensuring that it benefits all of humanity. The development of AI is rapidly progressing.  We must address the ethical implications of AI to ensure its responsible use.
    This example article discusses the impact of AI on various sectors.  It highlights both the benefits and challenges associated with AI. The future of AI depends on careful planning and ethical guidelines.
    """

    summary = summarize_article(article, num_sentences=2)  # Generate a 2-sentence summary

    print("Original Article:")
    print(article)
    print("\nSummary:")
    print(summary)
```

Key improvements and explanations:

* **Clearer Structure:**  The code is now structured into logical sections (Preprocessing, Frequency Calculation, Scoring, Selection, Combination) with comments explaining each step. This significantly improves readability.

* **Docstring:**  A docstring is included for the `summarize_article` function, explaining its purpose, arguments, and return value.  This is crucial for good documentation.

* **NLTK Resource Download:** The code now includes comments on how to download the necessary NLTK resources (punkt and stopwords).  Critically, these are commented out so the code *doesn't* automatically attempt to download the resources every time it runs.  The user is instructed to uncomment and run them *once* to install the necessary data. This avoids errors if the user doesn't have the data and makes the program more user-friendly.

* **Stop Word Removal:** Explicitly removes common English stop words using NLTK's `stopwords.words('english')`. This filters out words like "the," "a," "is," etc., which don't contribute much to the meaning of the article.  Converting words to lowercase before stop word removal is also crucial.

* **Punctuation Removal:** Removes punctuation marks to avoid them being counted as important words.

* **Word Tokenization and Filtering:** Combines tokenization and filtering into a single, more efficient list comprehension.  The `isalnum()` check ensures that only alphanumeric words are considered, removing stray punctuation marks or special characters that might have slipped through.  This is important for accurate frequency counting.

* **Frequency Distribution:** Uses `nltk.probability.FreqDist` for efficient frequency counting.

* **Sentence Scoring:**
    * **Lowercasing:** Converts words to lowercase during sentence scoring to ensure that "AI" and "ai" are treated as the same word.
    * **Sentence Length Penalty (Optional):** A conditional statement `if len(sentence.split(' ')) < 30:` is included to penalize very long sentences. This can help prevent the summarizer from selecting overly lengthy and rambling sentences.  This is optional but can improve summary quality.  The value of `30` is arbitrary and can be adjusted based on the typical sentence length in your articles.
    * **Clarity:** Improves the logic for incrementing sentence scores.

* **Sentence Selection with `heapq`:** Uses `heapq.nlargest` to efficiently select the top `num_sentences` sentences with the highest scores.  `heapq` is generally more efficient than sorting the entire `sentence_scores` dictionary, especially when selecting a small number of top elements.

* **Summary Combination:** Joins the selected sentences with spaces to create a coherent summary.

* **Example Usage with `if __name__ == "__main__":`:**  Includes an example usage section that demonstrates how to use the `summarize_article` function.  The `if __name__ == "__main__":` block ensures that the example code only runs when the script is executed directly, not when it's imported as a module.

* **Clear Output:** Prints both the original article and the generated summary for easy comparison.

* **Handles Empty Articles:**  The code now implicitly handles empty articles because if the article is empty,  the lists `sentences` and `words` will be empty, leading to an empty summary, without throwing errors.

* **Error Handling (Potential Future Improvement):** While not implemented here for brevity, you *could* add `try...except` blocks to handle potential `nltk` errors (e.g., if a resource is not found).

* **Adjustable Summary Length:**  The `num_sentences` parameter allows you to easily control the length of the generated summary.

This revised version is significantly more robust, readable, and efficient than the previous one. It follows best practices for Python code and provides a solid foundation for building a more sophisticated article summarizer.  It also avoids the common pitfalls of text processing, such as ignoring stop words or failing to lowercase text for consistent comparison.  The code is also now self-contained and runnable, without requiring external files or configurations (other than the initial NLTK resource downloads).