What it does
Fetches public HTML pages, sitemaps, and RSS/Atom feeds to build our search index. It does not submit forms or interact with your site.
Niriv Crawler
Meet NirivBot
NirivBot is the automated crawler that powers Niriv Search. It discovers, fetches, and indexes public web pages so they can be found by people searching in Nepali.
Identifying NirivBot in your logs
Every request from our crawler sends the following user-agent string. The
+https://niriv.com.np/bot
points back to this page.
NirivBot/1.0 (+https://niriv.com.np/bot)
How NirivBot behaves
Polite, predictable, and respectful of your settings.
Crawl rate
Requests are spaced out and rate-limited per host. NirivBot honors Crawl-delay and never hammers a single server.
Respects robots.txt
NirivBot reads and obeys your robots.txt and noindex/nofollow directives before and during crawling.
Privacy
Only publicly available content is stored. We do not collect personal data through crawling.
Technical details
How NirivBot discovers and indexes content.
Discovery
NirivBot discovers pages through sitemaps, RSS/Atom feeds, and links found on already-indexed pages. Submitting a sitemap via the Niriv Console helps us discover your content faster.
What we index
We index HTML content, meta descriptions, Open Graph data, and structured data (schema.org). We do not index password-protected pages, submitted form data, or non-public content. File types such as PDF, images, and video metadata may be extracted when available.
Recrawl frequency
The recrawl schedule depends on the site's popularity, update frequency of its content, and our crawl capacity. Most sites are revisited every few weeks. You can check your site's crawl interval on the Niriv Console after adding and verifying your domain.
Verification for site owners
To see crawl stats, request re-indexing, or manage how your site appears in search results, verify your site ownership on the Niriv Console. Verification is done by adding a meta tag to your site's homepage <head>. The Niriv Console provides the exact tag to insert.
How to control or block NirivBot
If you prefer not to be crawled, add the following to your site's robots.txt:
User-agent: NirivBot
Disallow: /To allow crawling of everything except a specific folder:
User-agent: NirivBot
Allow: /
Disallow: /private/Changes are picked up on the crawler's next visit. For urgent removal of a page from search, use the contact form.
Frequently asked questions
Common questions about NirivBot and the search index.
How do I verify my site?
Go to the Niriv Console and add your domain, then add the provided meta tag to your site's homepage <head>. Once verified, you can view crawl stats, adjust settings, and request re-indexing.
How do I request removal of a page?
For quick removal, use our contact form with the page URL. For ongoing control, add noindex meta tag or block in robots.txt — NirivBot will respect it on the next crawl.
Does NirivBot support nofollow?
Yes. NirivBot respects rel="nofollow" on links and meta robots directives including nofollow, noindex, nosnippet, and noarchive.
How can I request re-indexing of my site?
Verify your site on the Niriv Console, then use the crawl button to trigger an immediate re-crawl. Unverified sites are crawled on a best-effort schedule.
What IP addresses does NirivBot use?
NirivBot crawls from a set of dedicated IP addresses assigned to our servers. The range may change over time. For the most up-to-date list, contact us and we will provide the current addresses.
Questions about NirivBot?
We're happy to help with crawl issues, indexing requests, or verification. Reach out and a human from the Niriv team will respond.