With POPIA looming on the 1st of July 2021, I thought I'd discuss some of the tools we make use of at Intergreatme.
Security has always been at the forefront of our thinking when it has come to the design and implementation of technology, and from the start I tried to ensure that employees made responsible choices to the use of software services.
My preference has been to make use of tools that are open source, and that can be installed on our own server infrastructure. We make use of Microsoft Azure for our hosting as they were the first to market within the South African region (largely because some of the corporate customers we have also do not necessarily understand the implications of data transfer in terms of POPIA - but that's another story for another day!)
One of the products we have used pretty much from inception is Nextcloud. Originally, we made use of ownCloud, but later shifted to using Nextcloud. You can think of these products as an open source, host-it-yourself version of Dropbox. I simply created a VM, installed PHP and Maria DB (a fork of MySQL), installed Redis and set up Nextcloud. The configuration is pretty easy to make use of. I used Let's Encrypt to add SSL to the site. Let's Encrypt is a life saver when it comes to adding HTTPS to your web resources - especially when you are a fledgling start-up and do not have access to huge amounts of capital.
Nextcloud is pretty cool in that it also has a variety of apps that can be installed along side it which are accessible from inside the Web UI. We typically make use of Deck for Kanban style boards, which allows us to keep track of tasks.
On a few occasions, we have also made use of Talk, which provides Teams/Zoom type functionality from inside the browser, although our mileage has varied in terms of its ultimate success.
I also previously had an instance of OnlyOffice running which provided some collaboration tooling online between Nextcloud and Office productivity. We use Collabora Online at the moment, though I honestly preferred OnlyOffice - if only because it seemed to understand our regional settings better, and so opening up spread sheets and Word documents seemed to provide better compatibility for our region.
|The new Pettr logo
The logo draws on inspiration of the two animals from the survey - cats and dogs - and uses colours from the new brand palette he created for the project.
I have updated the favicon and added the logo to the website, and I am slowly working on some of the existing usability issues.
It has been an incredibly busy time for me for the past few weeks, and I have been trying to use my 'idle-time' in the evenings and on weekends to build some new functionality into Pettr.
There are three things I want to cover:
How things are calculated
I have had discussions around how the current costs are calculated, and how this information is presented by the Pettr calculator. The feedback is that people feel the averages do not accurately represent the cost of ownership.
And they are right, they don't. Why so?
The Pettr calculator works by grouping data from survey respondents that are allocated into particular bands, and then calculating averages from the various costs. It is not exactly difficult to calculate, although the primary calculator does come in at ~350+ lines of code. To optimise the process (and because it really isn't necessary), all calculations are pre-generated with the results being cached.
The current downfall of calculating pet affordability in this way is that the calculator does not differentiate a survey respondent that had three cats and two dogs, from a respondent that has two dogs that are both small, or any combination in-between. And because each individual will then have different costs because of the differences in how many animals they own, the averages can become skewed.
To cater for this, I have worked on a supplemental calculation that further breaks the current cost structure for bands of people where they only have a single pet (cat, small-sized dog, medium-sized dog, large-sized dog). It represents this data in the new feature: Breakdown.
The new breakdown feature is designed to make use of the new pre-calculated data where I have isolated survey respondents that responded where they own a single animal. The breakdown feature will allow users of the calculator to get a further break down based on the kind of animal they are interested in viewing the potential costs involved in owning that kind of animal.
Breakdown appears under the pie chart with Pet Choices, and allows you to select the type of animal that you want a breakdown for (assuming the data exists for that animal in that bracket).
Why results can differ
While working on the new Breakdown feature, I noticed that the affordability data represented as an average can sometimes be lower than the data shown in the new breakdown feature. And this was to be expected! In fact, it was deliberately catered for when the calculator is used where there is sufficient information available to provide information.
As seen above, a warning is displayed when there really isn't enough data to show give users accurate information.
Pettr deals with different people with different pet dynamics. An example of this situation is that in one of the bands, there is only survey respondent that has a cat. This respondent is actually spending more money on their pet than the average of all pet owners in that band.
Why would someone want to do this? A further look into their individual profile highlighted to me that this survey respondent spends above average on their cat's dietary requirements, and other needs because they only have the one pet, so feel they can spoil the animal more.
This is why it is so important to share the word of Pettr, and to get more people to participate in the survey. It is also why I am going to delay the release of this particular feature until I get more survey responses in.
Feature: How you compare
When you complete a survey in Pettr, you are currently presented with a success page, along with the ability to send your transaction information to your email address (and opt in to any email messaging I might want to provide in the future).
How you compare was designed to be a bit of a 'reward' for completing the survey, allowing you to see how you compare against the average in that affordability bracket.
I feel that Pettr has reached a point from a technological perspective where it has achieved my initial goal of being the world's first pet affordability calculator (that works off of crowd-sourced information).
And while there are plenty of websites that provide information about pet affordability, with my own research and initial survey design being oriented around publicly available information, I feel that they are either too US-oriented, or that they give bands of costs that do not really give people a clearer understanding of the costs they might spend when they get an animal.
The next few weeks will largely focus on getting the Pettr message out, and hopefully getting more survey responses in.
From a tech perspective, I am also going to be working on looking at individual pet breeds, and building data sets around each breed in order to help people make better choices about the animals they bring into their homes.
I am also acutely aware that the current iteration of Pettr has focused on 2020 within the bounds of both the survey, and the calculator. I want to do some strategic planning on how to either bring 2020's results into 2021, and also refocus on how I collect this information. Pettr might be a sideline project of mine, but I have a long list of things I want to learn, accomplish, and grow in this new venture.
This is just a space for me to add some of my thoughts on how to run these commands - and the might come in handy for other people too.
We are currently trying to import our current databases in to various nodes, but it keeps breaking (though most of the manuals and searches we've done seem to indicate that CSV is favoured over the PGDUMP method).
Creating a cockroach user
adduser --system --disabled-password --disabled-login --no-create-home --shell /bin/false --group cockroach
Creating the cockroach.service file
This is for use with systemd. The manual says you can also download a file to cater for this using wget
wget -qO- https://binaries.cockroachdb.com/cockroach-v20.1.7.linux-amd64.tgz | tar xvz
But I find it easier to just use the file directly - because most of our server environment is setup the same - so it is easier to replicate to different nodes. The file belongs in /etc/systemd/system/
[Unit] Description=Cockroach Database cluster node Requires=network.target [Service] Type=notify WorkingDirectory=/var/lib/cockroach ExecStart=/usr/local/bin/cockroach start --certs-dir=/var/lib/cockroach/certs --advertise-addr=localhost --cache=.25 --max-sql-memory=.25 --storage-engine=pebble TimeoutStopSec=60 Restart=always RestartSec=10 StandardOutput=syslog StandardError=syslog SyslogIdentifier=cockroach User=cockroach [Install] WantedBy=default.target
It is also good to remember the different systemctl commands and journalctl -ex command to trouble shoot when things go wrong.
When rebooting the machine, ensure you've run sudo systemctl enable cockroach (assuming your service is cockroach.service in /etc/systemd/system) to ensure it automatically restarts when you reboot the machine.
Other things to consider
Changing the access permission for the key files (as an example)
sudo chmod 600 node.key
sudo chmod 600 client.node.key
And to recursively change the ownership (usually from the dir where your data files will be and your certs)
sudo chown -R cockroach: .
One of the areas that I got feedback with regard to Pettr was that the income and disposable income levels are too high for certain countries, like the United Kingdom. Last night, I set out to tackle this problem and released the new changes.
In the first iteration of the user interface (UI), I designed the select boxes to just change the currency symbol instead of trying to alter the actual bands. I mean, the goal was to get something out quickly, so there are bound to be oversights in the design.
What this meant for users is that a lot of people visiting Pettr from these kinds of geographic areas would in all likelihood select a very low value - given the ranges are in large bands. And this would have affected their choices, but also skewed the overall scores because these countries 'buckets' of information would have most people falling into the same bucket instead of having a range to work from.
Before the changes, every
After the changes, bands
With the latest release, not only are the bands more dynamic, but I can set them individually per-country. While I have not had time to break each country into better income bands, this is the first step in doing so.
With this new enhancement, I have shifted from using a file-based config for countries and currency information, and have pushed all this data into CockroachDB. I've also duplicated these results in Redis (which I also use to store aggregated calculation data per-region) to reduce the number of calls to the database - there are actually very few calls to the database in general.
I will be rolling out further enhancements to the tech platform as it matures.
I haven't had too much feedback on the UI in general, other than people seem to think that the Pettr calculator is 'recommending' a particular kind of pet to them, where it is actually just showing them what is the most popular animal in the category.
Right now, I am not too concerned with that element, but I want to focus on breaking the data boundaries down a little further. People have provided great insight into their pets from the get-go, and it has been fantastic to see data come in, which presents an opportunity to sit back and investigate what can be done to further enhance the solution.
Edit - 05/10/2020
At first I didn't realise that my bands were slightly off (based on the way I was doing the calculations) where the lower limit and upper limit had a variance between them. I have since fixed the bands so that the end of the one band is -1 of the start of the next band.
It has been just over one week since I wrote up about Pettr, so I thought I would share some insights since the project was released to the public.
Firstly, wow! I cannot believe the constructive feedback I have received and the words of encouragement from people around my "pet project" - excuse the pun! Thank you to everyone that took the time to read the post, participate in the survey, and share the post in their social circles.
I am happy to report that there have been quite a few people who have actively participated in the survey, with results coming in from South Africa (obv. the easiest since it is where most of my network resides), the United Kingdom, and Australia. Early days, but I am happy with the progress so far.
I've been using a private-install of Matomo to track usage and conversions - opting in for a more privacy-centric analytics platform where the usage information isn't visible (or available) to a third party.
Getting the word out
The main route to gaining some traction has been through social media - it is, of course, one of the best ways to get instant visibility of a product.
I wrote the initial blog post and had that go off on a scheduled timer for Monday, 21 September 2020. I then created a basic post on LinkedIn, which up to now has had over 1000 views!
I also added a post to Facebook (a rarity for me), in order to further distribute the news. I also went ahead and wrote a WhatsApp message which I sent to people I know who have pets. In some cases, these friends also passed on the message to other people, helping send the message out further.
Here is the visitor map and no. of visits to Pettr since its 'social media release' last week.
Two days after releasing Pettr, I received a mail from a company asking about how we might be able to collaborate! My initial thoughts were: what the hell?! This idea-turned-project is just a few days old in terms of being released to the World, and already its gained some interest by big-business!
I can't say who it is, or what has been discussed - but I can say I am extremely excited for some of the potential ideas that have been discussed and how Pettr can continue to grow. I am feeling completely overwhelmed in the way that people have seen this project of mine and are looking for ways to help create an impact in what I am trying to achieve!
As Intergreatme takes priority, making changes to the Pettr platform will be gradual. Right now, my core focus is on bug fixes and minor enhancements (see below).
I have had some really fruitful discussions with people around possible ideas that are definitely great enhancements to the platform and will be working on these. I also think it is important to make it known that there is a development road map along with a product road map - so while Pettr is just starting out and in its infancy, not all ideas and suggestions might get implemented (I am also a one-man-band in this project that I've built entirely from scratch).
As always, feel free to share feedback, and if you have a pet, participate in the survey. I am really excited to see where this idea can go!
Bug fixes and patches (the tech stuff)
I identified an issue in the routine that converts survey results into calculation results, where I did not add initialisation variables in one of the loops which has resulted in some scores from South Africa 'leaking' into scores for other countries.
This has been patched, and I have generated the correct scores for surveys.
Links on Facebook add the awful ?fbclid GET parameter that FB appends to the URL in order to allow then to track your usage through other sites and tracking platforms. I currently use some GET param in the URL which was causing the system to throw error pages. I have written code that checks for the FB param in the URL and ignores it.
Fixed a logic flaw with the generation of the Pettr Survey GUID which is used while a user goes through the survey. In some instances, it was incorrectly generating a new GUID per-page which could have caused error pages to display, or issues while trying to complete a survey.