o ͿSi&@sddlZddlZddlmZmZddlmZmZmZddlZddl m Z ddl m Z ddl mZddlmZee Zejejd eeZGd d d Zd eed eefddZd eefddZd eefddZddZedkr{edSdS)N)datetime timedelta)TupleListDict) BeautifulSoup)OpenAI) load_dotenv)BearishDivergenceStrategy)levelc@seZdZddZdeeeeefeeeefffddZdedeefddZ deeeeeffd d Z dedeefd d Z e d edefddZ dS)TopMoversFetchercCs ||_dS)N)db)selfrr;/var/www/html/Trade-python/app/v1/services/zerolive/list.py__init__s zTopMoversFetcher.__init__returnc szh|jdddi}|r#t|dtddkr#|d|dfWS|d }|d }td d d |jd }|||fd d|D}fdd|D}|jdjddid||tdidd||fWSt y}zt j d|ddggfWYd}~Sd}~ww)z6Fetch top gainers and losers (with caching & mapping). et_moverstype top_movers last_updated)minutesgainerslosers top-gainersz top-losersdummy)api_key api_secret access_tokenrc"g|] }||tdqS)raw_name nse_symbol mapped_atgetrnow.0raw symbol_maprr -z5TopMoversFetcher.fetch_top_movers..cr!r"r&r)r,rrr.1r/z$set)rrrT)upsertzError in fetch_top_movers: exc_infoN) rfind_onerr(r _scrape_listr map_et_to_zerodha update_one Exceptionloggererror)rcached raw_gainers raw_losersstrategyrrerr,rfetch_top_moverss<      z!TopMoversFetcher.fetch_top_moversslugc Csd|}ddi}zLtj||dd}|t|jd}|d}g}|D]$}|d} | rI| d rIz ||| d Wq%t yHYq%wq%|rO|WSt d Wnt ym} z t d | WYd } ~ nd } ~ ww| |S) z Fetch the given ET page (gainers or losers) and extract the raw names. slug should be "top-gainers" or "top-losers". 8https://economictimes.indiatimes.com/stocks/marketstats/z User-AgentzuMozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36 )headerstimeout html.parser#table tbody tra.MarketTable_ellipses__M8PxMhrefzaTopMoversFetcher._scrape_list: HTTP scrape returned no symbols; falling back to headless browser.zXTopMoversFetcher._scrape_list: HTTP fetch failed (%s); falling back to headless browser.N) requestsr'raise_for_statusrtextselect select_oneappend_extract_symbol_from_hrefr7r8warning_scrape_list_via_selenium) rr@urlrCrespsouprowssymbolsrowar>rrrr4Bs@       zTopMoversFetcher._scrape_listcCs|d}g}||fS)z Legacy helper to load the ET gainers page and grab gainers. Currently this is implemented via the same HTTP + BeautifulSoup approach as `_scrape_list`, and returns (raw_gainers, losers=[]). r)r4)rr;rrrr_scrape_moversrs zTopMoversFetcher._scrape_moversc Cszddlm}ddlm}WntytdgYSwd|}|}z|dWnty<|dYnw|d|d |j |d }z~zQ| |t d t |jd }|d }g} |D]$} | d} | r| drz | || dWqhtyYqhwqh| WWz|WStyYSwty} z tjd| ddgWYd} ~ Wz|WStyYSwd} ~ wwz|WwtyYww)aFBest-effort fallback using a headless Chrome browser. This is only used if the primary HTTP+BeautifulSoup path fails or returns no symbols. If Selenium or a compatible driver is not installed on the server, this will log an error and return an empty list rather than crashing the app. r) webdriver)OptionszqTopMoversFetcher._scrape_list_via_selenium: Selenium not installed; cannot perform browser-based fallback scrape.rAz--headless=newz --headlessz --disable-gpuz --no-sandbox)optionsrErFrGrHzFTopMoversFetcher._scrape_list_via_selenium: headless scrape failed: %sTr1N)seleniumrZ!selenium.webdriver.chrome.optionsr[ ImportErrorr8r9 add_argumentr7Chromer'timesleepr page_sourcerLrMrNrOquit) rr@rZr[rRoptsdriverrTrUrVrWrXr>rrrrQ}sv                   z*TopMoversFetcher._scrape_list_via_seleniumrHcCs(|dd}|ddddS)uv '/glenmark-pharmaceuticals-ltd/stocks/companyid-4255.cms' → 'GLENMARK_PHARMACEUTICALS_LTD' z/stocksr/-_)splitupperreplace)rHpartsrrrrOsz*TopMoversFetcher._extract_symbol_from_hrefN)__name__ __module__ __qualname__rrrrstrr?r4rYrQ staticmethodrOrrrrr s.+0 ;r companiesrc Csd|}z*tjjjdd|dgdd}|jdjj}|dr't |WSt d |gWSt yI}zt d |gWYd }~Sd }~ww) z Use OpenAI to convert company names to NSE trading symbols (Zerodha KiteConnect style), returning a raw list of uppercase strings. zYou are a financial assistant. Convert the following Indian company names to their correct NSE trading symbols (used by Zerodha KiteConnect). Return ONLY a valid Python list of UPPERCASE strings. Company Names: z gpt-4o-miniuser)rolecontentg?)modelmessages temperaturer[z!Unexpected format from OpenAI: %szOpenAI API call failed: %sN)clientchat completionscreatechoicesmessagerystrip startswithevalr8debugr7r9)rvpromptresponseryr>rrrmap_company_to_symbols(     rcCsptdt|}|\}}g}|D]}|dp |dp d}|r,||qtdt|||S)uCHelper to return just the valid NSE symbols from today’s gainers.uFetching top gainers…r$r#z5Found %d gainer symbols (including raw fallbacks): %s r8infor r?r'rrnrNlen)rfetcherrrlrVgsymrrrget_top_gainer_symbolss    rcCsptdt|}|\}}g}|D]}|dp |dp d}|r,||qtdt|||S)NuFetching top losers…r$r#rz4Found %d loser symbols (including raw fallbacks): %sr)rrrlrrVlrrrrget_top_loser_symbolss    rcCstddlm}|j}tdt|}t}|\}}tdt|ddt|dt|dtd |d S) u Standalone debug entrypoint — requires a `db` argument or you can stub one in if you just want to test scraping. r) MongoClientz=== ET Top Movers Fetcher ===z Fetched in z.2fus → z gainers, z loserszGainers:N)pymongortestprintr rcr?r)rrrstartrrrrrmains  .r__main__)loggingrcrrtypingrrrrIbs4ropenairdotenvr r=r r~ basicConfigINFO getLoggerrqr8r rtrrrrrrrrs,     0