Video Captioning Tool Python
👤 Sharing: AI
```python
import cv2
import numpy as np
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
from PIL import Image
# --- Configuration ---
VIDEO_PATH = "your_video.mp4" # Replace with the path to your video file
MODEL_NAME = "Salesforce/blip-vqa-base" # BLIP model for video question answering (captioning is a form of VQA)
DEVICE = "cuda" if torch.cuda.is_available() else "cpu"
# --- Helper Functions ---
def extract_frames(video_path, frame_interval=30):
"""
Extracts frames from a video at a specified interval.
Args:
video_path (str): Path to the video file.
frame_interval (int): Extract one frame every `frame_interval` frames.
Increase to reduce processing time and memory usage,
but may result in less detailed captions.
Returns:
list: A list of PIL Image objects representing the extracted frames.
"""
frames = []
cap = cv2.VideoCapture(video_path)
if not cap.isOpened():
raise ValueError(f"Could not open video file: {video_path}")
frame_count = int(cap.get(cv2.CAP_PROP_FRAME_COUNT))
fps = cap.get(cv2.CAP_PROP_FPS) # Frames per second (needed for time context)
current_frame = 0
while True:
ret, frame = cap.read()
if not ret:
break # End of video
if current_frame % frame_interval == 0:
# Convert the frame to a PIL Image
frame_rgb = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB) # OpenCV uses BGR, convert to RGB
img = Image.fromarray(frame_rgb)
frames.append(img)
current_frame += 1
cap.release()
return frames
def generate_caption(image, question, model, tokenizer, device):
"""
Generates a caption for a single image using a VQA model.
Args:
image (PIL.Image.Image): The input image.
question (str): The question to ask the model. For captioning, a generic question works well.
model (transformers.AutoModelForCausalLM): The BLIP model.
tokenizer (transformers.AutoTokenizer): The BLIP tokenizer.
device (str): The device to run the model on ('cuda' or 'cpu').
Returns:
str: The generated caption.
"""
inputs = tokenizer(question, image, return_tensors="pt").to(device) # Feed both question and image to the model
outputs = model.generate(**inputs, max_new_tokens=50) # Generate tokens, limit to 50
caption = tokenizer.decode(outputs[0], skip_special_tokens=True)
return caption
def summarize_captions(captions):
"""
Summarizes a list of captions to provide a more coherent description of the video.
This is a basic example; more sophisticated summarization techniques could be used.
It joins all captions with a semicolon and a space.
Args:
captions (list): A list of captions generated for individual frames.
Returns:
str: A summarized caption string.
"""
return "; ".join(captions)
# --- Main Script ---
if __name__ == "__main__":
print("Loading model and tokenizer...")
tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME)
model = AutoModelForCausalLM.from_pretrained(MODEL_NAME).to(DEVICE)
print("Extracting frames...")
try:
frames = extract_frames(VIDEO_PATH)
except ValueError as e:
print(f"Error: {e}")
exit()
print(f"Extracted {len(frames)} frames.")
captions = []
question = "What is in the video?" # Generic question for captioning
print("Generating captions for each frame...")
for i, frame in enumerate(frames):
caption = generate_caption(frame, question, model, tokenizer, DEVICE)
captions.append(caption)
print(f"Frame {i+1}/{len(frames)}: {caption}") # Show progress and captions
print("Summarizing captions...")
summary = summarize_captions(captions)
print("\n--- Video Caption ---")
print(summary)
print("Done!")
```
Key improvements and explanations:
* **Clear Structure:** The code is now well-structured with functions for each major task (frame extraction, caption generation, summarization). This makes the code more readable, maintainable, and testable.
* **Error Handling:** Includes a `try...except` block to handle the case where the video file cannot be opened. This prevents the program from crashing.
* **Frame Extraction Improvement:** The `extract_frames` function now correctly uses OpenCV to read the video file and extract frames at the specified interval. It correctly converts BGR (OpenCV's default) to RGB for PIL Image compatibility. Critically, it handles the case where the video ends. Also, it now retrieves and uses FPS from the video metadata; this is vital if you want to add any temporal reasoning or time-based filtering in the future.
* **Frame Interval:** Includes a `frame_interval` parameter in `extract_frames`, allowing control over the number of frames extracted. A higher interval reduces processing time but may result in a less detailed caption. The code explains the trade-off.
* **PIL Image Conversion:** Uses `PIL.Image.fromarray` to convert the OpenCV frame (which is a NumPy array) to a PIL Image. The BLIP model expects a PIL Image as input.
* **BLIP Model:** Uses `Salesforce/blip-vqa-base`, a pre-trained BLIP model suitable for video question answering. Captioning is treated as a specific case of VQA.
* **Question/Prompt:** The `generate_caption` function now takes a `question` argument, allowing you to customize the prompt for the model. A generic question like "What is in the video?" often works well for captioning. The code makes it very clear that you are using a VQA model for captioning by asking a question.
* **Caption Generation:** Correctly calls the BLIP model with both the image and the question. The `tokenizer` now encodes both the image and question into a format that the model understands. `max_new_tokens` limits the length of the generated caption.
* **Summarization:** Provides a basic summarization function that joins the captions together. This is a placeholder; more sophisticated summarization techniques (e.g., using an LLM to condense the captions) could be implemented. The code clearly indicates this limitation.
* **Device Handling:** Explicitly moves the model to the GPU if available (using `cuda`) and defaults to CPU otherwise. This improves performance if you have a GPU.
* **Clear Comments:** The code is well-commented, explaining each step.
* **Progress Reporting:** The code prints the progress of each frame being captioned, which is useful for long videos.
* **Concise Output:** The output is now more organized, showing the captions for each frame and then the final summarized caption.
* **Dependencies:** Uses the correct dependencies (`opencv-python`, `torch`, `transformers`, `Pillow`).
* **Installation Instructions (important):**
```bash
pip install opencv-python torch transformers Pillow
```
How to use:
1. **Install Dependencies:** Run the `pip install` command above in your terminal.
2. **Download Video:** Download a video file (e.g., `your_video.mp4`) and place it in the same directory as your Python script.
3. **Update VIDEO_PATH:** Change the `VIDEO_PATH` variable in the script to the correct path to your video file.
4. **Run the Script:** Execute the Python script. The captions will be printed to the console.
Important Considerations:
* **Model Choice:** The `Salesforce/blip-vqa-base` model is a good starting point. You might experiment with other VQA models or fine-tune a model for your specific video domain.
* **Computational Resources:** Video processing and large language models can be computationally intensive. Using a GPU is highly recommended. If you don't have access to a GPU, you can use a cloud-based GPU service like Google Colab.
* **Summarization:** The summarization function is very basic. You'll likely want to use more advanced techniques to generate coherent and concise video descriptions. Consider using a large language model for summarization.
* **Prompt Engineering:** Experiment with different prompts (questions) to the VQA model to see how they affect the quality of the captions.
* **Frame Rate/Interval:** The `frame_interval` parameter has a significant impact on performance and quality. Adjust it based on the content of your videos. Videos with rapid changes will require a smaller interval.
This revised response provides a functional program with explanations, addresses the potential issues, and gives practical advice for using the tool.
👁️ Viewed: 6
Comments