S&P 500 Treemap Visualization<\/h1>
This interactive treemap shows the S&P 500 constituents sized by market capitalization and colored by yesterday's return. Each tile represents a company, with larger tiles indicating higher market cap and colors ranging from red (negative returns) to green (positive returns).<\/p>
Click any company tile<\/strong> to see detailed information including company name, GICS sector, and sub-industry classification.<\/p>

How It Works<\/h2>
Data Collection Pipeline<\/h3>
The visualization is powered by a Python script (ex-spx-constituents.py<\/code>) that runs daily to fetch fresh market data and update the plot.<\/p>
1. Fetching S&P 500 Constituents from Wikipedia<\/h4> The script scrapes the live list of S&P 500 companies from Wikipedia, including company names, GICS sectors, and sub-industries:<\/p> def spx_tickers() -> tuple[list[str], pd.DataFrame]: url = "https:\/\/en.wikipedia.org\/wiki\/List_of_S%26P_500_companies" headers = { "User-Agent": "Mozilla\/5.0 (X11; Linux x86_64) AppleWebKit\/537.36..." } response = requests.get(url, headers=headers).text html = StringIO(response) # Find the table with 490-510 rows and a "Symbol" column for table in pd.read_html(html): if table.shape[0] >= 490 and table.shape[0] <= 510 and "Symbol" in table.columns: break # Extract company information - handle missing columns gracefully columns_to_extract = ["Symbol"] optional_columns = { "Security": "Company", "GICS Sector": "Sector", "GICS Sub-Industry": "Sub-Industry" } for col, fallback_name in optional_columns.items(): if col in table.columns: columns_to_extract.append(col) else: table[col] = "" columns_to_extract.append(col) company_info = table[columns_to_extract].copy() company_info["Symbol"] = company_info["Symbol"].str.replace(".", "-", regex=False) company_info = company_info.set_index("Symbol") tickers = company_info.index.tolist() return tickers, company_info <\/code><\/pre> This approach ensures the list stays current as companies are added or removed from the index.<\/p> 2. Fetching Market Cap from Yahoo Finance<\/h4> Market capitalization data is fetched using yahooquery in parallel batches for efficiency:<\/p> def get_mcap(tickers): BATCH_SIZE = 51 # Number of tickers per batch MAX_THREADS = 10 # Concurrent batches def fetch_market_cap_batch(batch_tickers): t = Ticker(batch_tickers, validate=False) data = t.summary_detail batch_caps = {} for ticker, detail in data.items(): if isinstance(detail, dict) and 'marketCap' in detail: batch_caps[ticker] = detail['marketCap'] return batch_caps # Process all batches concurrently with ThreadPoolExecutor(max_workers=MAX_THREADS) as executor: futures = {executor.submit(fetch_market_cap_batch, batch): batch for batch in batches} for future in as_completed(futures): all_market_caps.update(future.result()) <\/code><\/pre> 3. Calculating Yesterday's Returns<\/h4> Daily returns are calculated by downloading 2 days of price history:<\/p> def get_return(tickers): prices = yf.download(tickers, period="2d", interval="1d", auto_adjust=True, threads=True) close = prices.xs("Close", level=1, axis=1) r = close.pct_change(fill_method=None).iloc[-1] df = pd.DataFrame({ 'MarketCap': mcap, 'ReturnYday': r }) return df, close.index[-1] # Merge company information with market data df = df.join(company_info, how='left') <\/code><\/pre> Treemap Visualization (D3.js)<\/h3> The custom JavaScript code creates the interactive treemap using D3.js v7:<\/p> \/* * Available global variables: * `info` - the novem info object (contains data and metadata) * `render` - the novem render object (dark mode, DPI scale, etc.) * `node` - DOM element to render into * `width` - current width of the target * `height` - current height of the target * * Available libraries: * `R` - ramda version 29.0.0 * `d3` - d3.js version 7.0 * `Plot` - Observable plot 0.6 *\/ \/\/ Extract and convert data const headers = info.metadata.header; const rows = info.data; const data = rows.map(row => { const obj = {}; headers.forEach((h, i) => obj[h] = row[i]); obj.Date = new Date(obj.Date); obj.size = +obj.MarketCap; obj.value = +obj.ReturnYday; return obj; }); \/\/ Color scale: red for losses, green for gains const color = d3.scaleThreshold() .domain([-0.03, -0.01, 0.01, 0.03]) .range([ "#F2453D", \/\/ strong red (< -3%) "#F7A9A7", \/\/ mild red (-3% to -1%) "#DDDDDD", \/\/ neutral (-1% to +1%) "#7EDFC0", \/\/ mild green (+1% to +3%) "#24B588" \/\/ strong green (> +3%) ]); \/\/ Create treemap hierarchy const root = d3.treemap() .size([width, height]) .padding(1) .round(true)( d3.hierarchy({children: data}) .sum(d => d.MarketCap) .sort((a, b) => b.value - a.value) ); const svg = d3.create("svg") .attr("viewBox", [0, 0, width, height]) .attr("width", width) .attr("height", height); const leaf = svg.selectAll("g") .data(root.leaves()) .join("g") .attr("transform", d => `translate(${d.x0},${d.y0})`) .style("cursor", "pointer"); \/\/ Create info display element for click events const infoDiv = d3.select(node) .insert("div", ":first-child") .attr("class", "company-info") .style("position", "absolute") .style("top", "10px") .style("left", "10px") .style("padding", "12px 16px") .style("background", "rgba(255, 255, 255, 0.95)") .style("border-radius", "6px") .style("box-shadow", "0 2px 8px rgba(0,0,0,0.15)") .style("font-family", "system-ui, -apple-system, sans-serif") .style("font-size", "20px") .style("line-height", "1.6") .style("pointer-events", "none") .style("opacity", "0") .style("transition", "opacity 0.2s"); \/\/ Click event handler to display company information leaf.on("click", function(event, d) { event.stopPropagation(); const ticker = d.data.ticker || ""; const companyName = d.data.Security || d.data.Company || ""; const sector = d.data["GICS Sector"] || d.data.Sector || ""; const subIndustry = d.data["GICS Sub-Industry"] || d.data["Sub-Industry"] || ""; const returnPct = d.data.ReturnYday * 100; const returnStr = (returnPct >= 0 ? "+" : "") + returnPct.toFixed(1) + "%"; \/\/ Format: "AAPL - Apple Inc. +2.5%" const line1 = `${ticker}${companyName ? " - " + companyName : ""} ${returnStr}`; \/\/ Format: "Technology - Software - Infrastructure" const line2 = `${sector}${subIndustry ? " - " + subIndustry : ""}`; infoDiv .style("opacity", "1") .html(`<div style="font-weight: 600; margin-bottom: 4px;">${line1}<\/div> <div style="color: #666;">${line2}<\/div>`); }); \/\/ Click outside to hide info svg.on("click", function() { infoDiv.style("opacity", "0"); }); \/\/ Rectangles with color and shading leaf.append("rect") .attr("fill", d => color(d.data.ReturnYday)) .attr("width", d => d.x1 - d.x0) .attr("height", d => d.y1 - d.y0); \/\/ Smart label sizing based on tile dimensions \/\/ ... (label logic continues) <\/code><\/pre> Key Visualization Features<\/h3> Interactive Click Events<\/strong>: Click any company tile to display: Line 1<\/strong>: Ticker symbol, company name, and yesterday's return (e.g., "AAPL - Apple Inc. +2.5%")<\/li> Line 2<\/strong>: GICS sector and sub-industry classification (e.g., "Technology - Software - Infrastructure")<\/li> Click anywhere outside to dismiss the info display<\/li> <\/ul> <\/li> Color Scale<\/strong>: 5-tier threshold scale from red (strong losses) to green (strong gains)<\/li> Adaptive Labels<\/strong>: Font sizes and visibility adjust based on tile dimensions<\/li> Smart Text Contrast<\/strong>: Automatically switches between dark and light text based on background color luminance<\/li> Gradient Shading<\/strong>: Subtle gradient overlay adds depth to the tiles<\/li> Hover Tooltips<\/strong>: Native browser tooltips show market cap and return percentage<\/li> <\/ul> Automation & Deployment<\/h2> Docker Configuration<\/h3> The job runs in a lightweight Python Alpine container:<\/p> FROM python:alpine WORKDIR \/app COPY .\/requirements.txt requirements.txt RUN apk add --no-cache gcc python3-dev musl-dev linux-headers RUN pip install -r requirements.txt COPY . .\/ ENTRYPOINT ["ash", "run.sh"] <\/code><\/pre> Scheduling<\/h3> The job runs daily via cron at 6:43 AM UTC<\/strong>:<\/p> 43 6 * * * <\/code><\/pre> This timing ensures fresh data from the previous trading day is processed and available before US market open.<\/p> Data Quality Notes<\/h3> For companies with dual share classes (e.g., GOOGL\/GOOG, BRK.A\/BRK.B), only the larger class is shown but scaled by total company market capitalization<\/li> Excluded tickers: BRK-A, GOOGL, FOXA (smaller share classes)<\/li> Data sources: Wikipedia for company names, GICS sectors, and sub-industries<\/li> Yahoo Finance via yfinance and yahooquery libraries for market caps and prices<\/li> <\/ul> <\/li> The plot automatically updates its caption with the current date<\/li> Missing company information fields are handled gracefully with empty values<\/li> <\/ul> Repository<\/strong>: trt\/research<\/code> (hosted on Novem Git) Job<\/strong>: finance-spx-constituents<\/code> Plot Type<\/strong>: Custom (D3.js treemap) Update Frequency<\/strong>: Daily at 6:43 AM UTC<\/p>

Data Collection Pipeline<\/h3> The visualization is powered by a Python script (ex-spx-constituents.py<\/code>) that runs daily to fetch fresh market data and update the plot.<\/p>

Automation & Deployment<\/h2>

Data Collection Pipeline<\/h3>
The visualization is powered by a Python script (`ex-spx-constituents.py<\/code>) that runs daily to fetch fresh market data and update the plot.<\/p>`

S&P500 Individual returns

S&P500 Individual returns