Multi‑source company intelligence platform – automated collection and AI‑powered analysis of company data from Crunchbase, LinkedIn, and Reddit.
- Aggregates company and social data from Crunchbase, LinkedIn, and Reddit.
- Automates web scraping via CrewAI agents and Bright Data MCP (handles JS rendering, proxy rotation, and CAPTCHAs)
- AI-powered analysis and insight generation with Gemini models
- Real-time visualization in a Streamlit dashboard
- Modular architecture for easy addition of new data sources or workflows
- Initiate: Data-retrieval with specialized CrewAI agents.
- Orchestrate: API calls & dynamic scraping with Bright Data MCP – handles JS, proxies, rate limiting, CAPTCHA, more.
- Normalize: Store and send data to Gemini AI for instant analysis.
- Display: Results and actionable insights in Streamlit dashboard.
- Language: Python 3
- Orchestration: CrewAI multi‑agent framework
- Protocol: Model Context Protocol (MCP)
- Scraping: Bright Data MCP & Web Scraper APIs
- AI: Google Gemini models
- Dashboard: Streamlit
| Source | Data type | Content | Integration |
|---|---|---|---|
| Crunchbase | Company profiles | Funding, team, metrics, news | 100% MCP |
| Professional data | Jobs, updates | Hybrid (MCP + API) | |
| Public sentiment | Discussions, opinions, reviews | 100% MCP |
- Competitive intelligence & company research
- Market trend analysis
- Real-time sentiment & topic monitoring
- Investment/M&A targeting
- Multi-source data discovery for analysts
- Python 3 & Node.js
- Bright Data account with API key
- PLUS Web Unlocker and Browser API zones
- Gemini API access
# Clone the repo
git clone https://github.com/brightdata/trendscan.git
cd trendscan
# Create and activate Python virtual environment
uv venv
# Install dependencies
uv pip install -r requirements.txt- Open the
.envfile in your project’s root directory. - Fill in your real credentials and API keys.
- (Optional) To use a different Gemini model, set
LLM_MODELin.env. See Gemini model docs.
.env Variables (example):
| Variable | Description |
|---|---|
| BRIGHT_DATA_API_KEY | Your Bright Data API Key |
| GEMINI_API_KEY | Your Gemini API Key |
| ... | ... |
Launch the Streamlit web interface with:
streamlit run streamlit_trendscan.pyYou’ll be able to initiate data retrieval, run the pipeline, and interact with the dashboard in your browser.
PRs, feedback, and feature suggestions are welcome!
- Open an issue for bugs and improvements.
- For large contributions, please open a discussion first.
Distributed under the MIT License.
Have fun scanning trends and supercharging company intelligence!