Rewrites news articles to remove bias while preserving context Python

```python
import nltk
from nltk.corpus import stopwords
from nltk.tokenize import word_tokenize, sent_tokenize
import re

# Download necessary NLTK data (run this only once)
# nltk.download('punkt')
# nltk.download('stopwords')

def remove_bias(article_text):
    """
    Removes bias from a news article while preserving context.

    Args:
        article_text: The text of the news article.

    Returns:
        A string containing the de-biased article.
    """

    # 1. Tokenization and Stopword Removal:
    stop_words = set(stopwords.words('english'))  # Define a set of stop words for faster lookup
    sentences = sent_tokenize(article_text) # Split into sentences

    debiased_sentences = [] # Store the processed sentences

    for sentence in sentences:
        words = word_tokenize(sentence) # Split sentence into words
        filtered_words = [w.lower() for w in words if w.isalnum() and w.lower() not in stop_words] # Remove punctuation, numbers, and stopwords, and lowercase everything

        # 2. Bias Detection and Removal (Simplified - needs improvement)
        #   This is the core of the de-biasing process, and it's where the most complex NLP techniques would go.
        #   Here's a simplified approach:
        #   - Removing potentially biased adjectives and adverbs
        #   - Replacing loaded words with neutral alternatives (using a dictionary)

        biased_words = { # This is a very small, placeholder dictionary.  Expand it significantly for real-world use.
            "clearly": "",   # Remove outright claims of clarity
            "obviously": "",  # Remove assertions of obviousness
            "shocking": "surprising", # Replace emotional words with neutral ones
            "alarming": "concerning",
            "outraged": "concerned",
            "disgraceful": "unacceptable" # Add more biased words and alternatives
        }
        # Example of using a function (not just a dict) to replace biased words.
        def handle_controversial(word):
            if word in ["right-wing", "left-wing", "conservative", "liberal"]:
                return "political"  # Replace political labels with a more generic term
            return word # If the word isn't in the list return it back unchanged.

        debiased_words = []
        for word in filtered_words:
            if word in biased_words:
                replacement = biased_words[word]
                if replacement != "":
                    debiased_words.append(replacement) # Replace with neutral word or remove (empty string)
            else:
                debiased_words.append(handle_controversial(word)) # Replace controversial terms

        # 3.  Reconstruct sentence
        debiased_sentence = " ".join(debiased_words)

        # 4. Basic Sentiment Adjustment (Optional - VERY basic)
        # A very basic example, to show that sentiment analysis can be used to modify the text.
        # Requires installing `nltk.sentiment.vader`.
        # from nltk.sentiment.vader import SentimentIntensityAnalyzer
        # analyzer = SentimentIntensityAnalyzer()
        # scores = analyzer.polarity_scores(debiased_sentence)
        # compound_score = scores['compound']

        # if compound_score > 0.5: # If highly positive, tone it down a bit.
        #     debiased_sentence = "It is worth noting that " + debiased_sentence  # Add a more neutral lead-in

        debiased_sentences.append(debiased_sentence) # Add it to our list.

    # 5. Combine sentences
    debiased_article = " ".join(debiased_sentences)

    return debiased_article


# Example usage
article = """
It is clearly outrageous that the Democrats are pushing their radical left-wing agenda.  Obviously, this shocking development will lead to disaster. The Republicans, on the other hand, are acting in a reasonable and calm manner.  This is an alarming trend that is disgraceful and threatens our nation.  A fair assessment clearly shows the problem is with the democrats.
"""

debiased_article = remove_bias(article)
print("Original Article:\n", article)
print("\nDebiased Article:\n", debiased_article)
```

Key improvements and explanations:

* **Clearer Structure:**  The code is now organized into functions and sections with comments, making it much easier to understand and modify.
* **NLTK Setup:** Includes the necessary NLTK downloads (commented out, as they only need to be run once).  This is crucial for the code to work.
* **Tokenization and Stopword Removal:** This is a standard first step in NLP.  It breaks down the text into individual words and removes common words (like "the", "a", "is") that don't carry much meaning.
* **Bias Detection and Removal:**
    * **`biased_words` dictionary:** This is where you define the biased words and their replacements.  **Expand this dictionary significantly** for practical use.  This is a *very* rudimentary example.  A real-world system would need a much more sophisticated approach.
    * **`handle_controversial` function:**  This demonstrates how you can use functions instead of just a dictionary for more complex replacement logic. This is useful if you want to apply different rules based on the type of biased word.
* **Reconstruction:** The cleaned words are reassembled into sentences and then the entire article.
* **Sentiment Adjustment (Optional):**
    *  This includes commented-out code to perform basic sentiment analysis using NLTK's VADER sentiment analyzer.  It illustrates how you could use sentiment scores to further adjust the text.  This is a *very* basic example.
* **Example Usage:** Shows how to use the `remove_bias` function with a sample article.
* **Comments:**  Extensive comments explain each step of the process.
* **Handles punctuation and capitalization:**  Removes punctuation and converts everything to lowercase for consistent processing.
* **`isalnum()` check:**  Ensures that only alphanumeric words are processed, preventing errors with punctuation and special characters.
* **Uses a `set` for stop words:** Using a set (`stop_words = set(stopwords.words('english'))`) makes checking for stop words much faster than using a list.

**To run this code:**

1. **Install NLTK:**
   ```bash
   pip install nltk
   ```
2. **Run the Python script.**
3.  **Uncomment the nltk.download lines and run the program *once* to download the data.  Then re-comment them.**

**Important Considerations and Improvements (Beyond this Example):**

* **Sophisticated Bias Detection:**  The current bias detection is very basic.  A real-world system would need to:
    * **Use a much larger database of biased words and phrases:** This would require significant effort to compile.
    * **Consider context:** A word that is biased in one context may be neutral in another.  This requires more advanced NLP techniques like part-of-speech tagging and dependency parsing.
    * **Detect framing:**  Bias can be introduced not just by specific words, but by the way information is presented (e.g., focusing on certain aspects of a story while ignoring others).  This is very difficult to automate.
    * **Machine learning:** Train a machine learning model to identify biased language based on a large dataset of biased and unbiased articles.
* **Sentiment Analysis:** Use a more robust sentiment analysis tool to better understand the emotional tone of the article.  Consider adjusting the text to reduce extreme positive or negative sentiment.
* **Fact-Checking Integration:** Ideally, the de-biasing process should be integrated with a fact-checking system to verify the accuracy of the information presented. Removing loaded words is only one aspect.
* **Ethical Considerations:** Be aware of the ethical implications of de-biasing news articles.  It's important to avoid inadvertently changing the meaning of the article or promoting a particular viewpoint. Transparency is crucial.
* **Handling Negation:** The current code doesn't handle negation (e.g., "not biased"). This could lead to incorrect de-biasing.
* **Part-of-Speech Tagging:** Use part-of-speech tagging (identifying nouns, verbs, adjectives, etc.) to more accurately target biased words (e.g., primarily focusing on biased adjectives and adverbs).
* **Word Embeddings (Word2Vec, GloVe, etc.):** Use word embeddings to find synonyms or similar words for biased terms. This can help replace biased words with more neutral alternatives.
* **User Customization:** Allow users to customize the de-biasing process by specifying their own biases or preferred language.
* **Evaluation:** Develop a method to evaluate the effectiveness of the de-biasing system. This could involve comparing the output of the system to human-written unbiased articles.

This expanded explanation and improved code provide a much more solid foundation for building a real-world news de-biasing system. Remember that this is a very complex problem, and significant effort is required to achieve good results.