Today marks the 5th anniversary of the launch of searchmysite.net, and it is also a year since the last blog entry (which was the Four year retrospective), so now seems a good time to summarise progress.

Repositioning searchmysite.net as a personal website search

The original objective of searchmysite.net was the ambitious “search just the good stuff” (it even indexed wikipedia until that was stopped to cut costs), then it was changed to “search non-commercial sites”, although few people seemed to know what “non-commercial sites” were so it was changed to “search personal and independent sites”. However, in order give the site more focus, and in line with the Unix philosophy of trying to “do one thing and do it well”, independent sites are now being removed. I’m really sorry to see some of my favourite independent sites go, but searching all good personal sites is hopefully a much more achievable goal for a spare-time project than searching all good personal and independent sites.

Some exciting news, first hinted at in the Three year retrospective: the searchmysite.net redesign has been launched, thanks to its first major open source contributor Lucas Gramajo.

First major open source contributor

Just to clarify, there have been other open source contributors to searchmysite.net before. The first was Binyamin, who contributed the favicon on 23 Oct 2020. And after searchmysite.net was fully open sourced in Dec 2020, there have been other contributions in the form of bug reports, discussion etc. Plus of course searchmysite.net is entirely based on other open source projects like Apache Solr, so without all the contributions to those great projects, this project wouldn’t exist.

This post is to provide an update on the automated SEO searches issue described in my last post Almost all searches on my independent search engine are now from SEO spam bots. It references the discussion on Hacker News (HN) from Mon 16 May.

Traffic and system performance and stability

In terms of traffic, there were 18,034 visitors to blog.searchmysite.net on Mon 16 May and 2,699 visitors to searchmysite.net, pretty much all of which came as a result of the HN post:

Introduction

It has been around 1.5 years since I launched searchmysite.net as a side-project to try and address the problems with the current commercial internet search offerings, and I reckon I’ve now spent around 650 hours working on it1. The 2021 year end seems a good opportunity for a retrospective of what has gone well, what has gone neither well nor not so well, and what has not gone so well. And based on that, some thoughts on where it should go in 2022. Given the nature of the project, this will be an open and honest account rather than a Silicon Valley style “fake it ’til you make it” piece.

searchmysite.net is an open source search engine and search as a service for personal and independent websites, which has a unique approach to advertising to try to tackle spam. I mentioned this approach to advertising in my last post. While this attracted a lot of positivity, it also unfortunately got some negativity1, so I thought I’d write a quick post to clarify my position on advertising and search engines, hopefully in positive terms.

Introduction

I used the quote “talk is cheap, show me the code” in the introduction to my first post on searchmysite.net. Well, here it is: https://github.com/searchmysite/searchmysite.net/.

Why aren’t other search engines open source?

Pretty much every other search engine treats their inner workings as a closely guarded secret. This is to stop the spammers figuring out how to game the system and increase the ranking of their results to earn a greater share of the advertising revenue. However, this isn’t a concern for searchmysite.net, because its operating model is designed to both keep spam out and to remove the financial incentive for spam in the first place:

Introduction

In my last major post, searchmysite.net update: Seeding and scaling from 25 Sept 2020, I concluded “I’ll ease off on enhancements, and try to focus on adoption for a while”. So how has that gone? Well, I had a nice burst of activity between 16 and 19 Oct, with 215 sites submitted in 4 days, which was great and led to some really useful feedback, including the first site to use the searchmysite.net API to power their search page. However, the higher levels of usage did expose some issues, which I’ll summarise here.

Welcome

Welcome to the first post on searchmysite.net’s new blog. Well the first one first posted on this blog - there are actually some earlier posts which I’ve copied over from my personal site where they were first published:

I decided to split the blogs out because there’s quite a bit I want to write on both, and searchmysite.net does seem to be starting to get a life of its own now.

Introduction

This post shows how I added a simple search page to my personal website with searchmysite.net. You can click on it and try it out via the Search link at the top of any page on my site. Note that it is very simple.

I know I don’t really need search functionality given that my personal website only currently has a few posts, but I wanted to test the process out and demonstrate it.