In this post, I’m going to try to explain what GPTs and LLMs are in the simplest terms possible. My hope is that, by the end, you’ll have a better understanding of what’s involved in these tools and what that means for AI’s practical use in everyday life and business. There will definitely be nuances and details missing, but I believe that grasping even just the basics of how ChatGPT works is much more helpful than the existential terror and misguided reverence it fosters now.

Terminology

I’ve already used two of the most relevant terms in the introduction, “LLM” and “GPT.” I also used “AI,” but I’m hoping that concept is already somewhat clear to you if you chose to read this. Computer software developers love collapsing phrases into initialisms like this to save time, but this often leaves people who aren’t already in the know very much out of the loop. In many cases, what the initialisms stand for doesn’t ultimately matter for most people, but in this case, I believe that knowing what LLM and GPT stand for and then learning what the phrases actually mean is more than half of what you need to really understand what’s going on at the heart of modern AI.

What is an LLM and Why is It Important?

“LLM” in the context of modern AI stands for “Large Language Model.” At its core, an LLM is a big (“large”) list of phrases and sentences (“language”) with some complicated math applied to rank how each word relates to other words in the list (“model”).

More specifically, an LLM is a storage space (“database”) containing as many sample documents as possible. Those documents are split out into individual phrases and complete sentences (“strings”) depending on a specified number of letters and punctuation (often resulting in millions to trillions of strings or more), then those strings are separated into individual words and analyzed. The analysis checks every word’s proximity to every other word in every string along with any punctuation used near those words, and it ranks the probability that one word is likely to appear next to another word in a particular order.

Additional data (like the topic, mood, tone, source, or any number of other ways to categorize the documents) can be provided before checking the strings in the database in order to further identify contexts where certain word combinations might appear. For example, it might be helpful to know when the sample string comes from someone’s social media post versus someone else’s academic journal article versus a movie review website. Typically, an LLM will cover a single context, e.g. only social media posts or only information covering a certain topic, because that kind of specialization reduces the work required to categorize the samples. ChatGPT’s LLM is exemplary in that the software developers have gone to great lengths to collect an unfathomable number of sample texts and meticulously categorize those samples in as many ways as possible.

What is a GPT and How Does It Work?

The “GPT” in “ChatGPT” stands for “Generative Pre-trained Transformer.” To oversimplify things a little, a GPT uses the words and proximity data stored in LLMs to determine what words would be most reasonable to place together in a given context (“pre-trained”) in order to re-combine those words (“transformer”) into hopefully different resulting text (“generative”).

What makes a GPT impressive is that it does the analysis and reorganization of words in both directions—you can write to a GPT, and it will inspect your writing (“input”) to find the most relevant word rankings within its LLM in relation to what you’ve written in order to determine the appropriate words to combine as a response (“output”). In addition to word rankings, a GPT will also attempt to determine the correct context to check word rankings within based on what you’ve written and where those combinations most commonly occur. The best ones will have a way to decide if a given input is you attempting to tell it what context to pull from, again all based on the previous categorizations and ranking of words provided by the LLM.

GPT programs that I’ve used often have a space where you can give it starting context to use before considering someone’s requested input. Every additional word added to the input is used to filter what sample text should be used, and many GPT chatbots provide that context before letting you write to it. This context is essentially added to every input to add extra filtering when generating output. Using these contexts can help ensure that output is created within a certain set of parameters and pre-filtered options, which is how the polite and subservient “personality” of ChatGPT is achieved.

It’s Not Intelligence

With the important definitions out of the way, it’s important to note that GPTs are not actually thinking or coming up with any unique thoughts. The only “intelligence” on display is the cleverness of using the word proximity data to produce output that sounds natural because the data samples were collected from natural writings. Some have cleverly suggested “Applied Statistics” as a new name for AI in reference to how GPTs construct output based on word proximity and positioning, but “AI” is simply too disseminated into the public. In my opinion, “Automated Inference” would be more fitting because it keeps the “AI” acronym while also painting a more accurate picture of what the program is actually doing. The LLMs give the GPTs the connections required between words, and that helps the program infer what its output should be.

Again, the most important takeaway here is that there’s no real thinking happening; GPTs are simply combining words. There is no analysis or understanding of subject matter. And while some GPTs store a short history of input and output to act as a simple working memory, it’s still just using the stored data as further filtering. In fact, when GPTs are not configured with enough memory, the results can be funny at best but frustrating at worst. The responses are often so natural that it’s easy to forget that it’s not actually thinking and keeping track of your whole “conversation” unless it’s repeatedly reminded of what has been said before as the stored memory is cleared out in favor of more recent input and output.

On top of it all, it’s still only able to combine words in ways that it has seen before from its LLM. While it’s possible for output to be arranged in interesting and unexpected ways, it is impossible for it to create something never seen before. (Though this article where the author co-creates a constructed language with ChatGPT (web archive) is quite an impressive manipulation of the available data in the LLM!)

Art Bots and How They’re Related

To me, art bots are the most impressive application of these generators. Instead of using collections of text, they instead use large models of images. Those images are categorized and collected in a similar way to LLMs, and then those images are chopped up and reorganized in ways it deems reasonable based on rankings of different parts of the images. This is impressive because while text is fairly straightforward for a computer to work with and find connections within, the mechanism for finding connections between elements of images is still fairly mysterious to me.

One thing I do know is that image GPTs utilize extensive cataloguing of their image sets, including sub-categorization of various elements within a single image. Early AI image identifiers attempted to match various elements of input images with other elements within its own data sets. Oftentimes, the data sets would not be big or varied enough to identify things in slightly different poses or angles than what it had in its model, which would lead to misidentification. But now with unfathomably more images in the available data set, it’s easier for matches to be found.

Extrapolating from this, an image GPT must use this identification strategy to piece together images from its data set and smooth them together using some kind of pre-defined method. When the number of available images is so vast and its cataloguing and organization so thorough, piecing together images that match an arbitrary input follows an extremely similar structure to regular text GPTs, except “proximity” now needs to take “up” and “down” into consideration in addition to “left” and “right” as with a sentence.

GPT AIs are Extremely Impressive and Useful

Even though I’ve broken down modern AI into its basic parts to help you understand how it works, it’s still important to recognize just how impressive the things that developers like OpenAI do with ChatGPT really are. They’re working with, analyzing, categorizing, and organizing gigabytes (possibly terabytes) of text samples that are processed and used based on prompts that anyone can supply. For reference, one letter is a byte and a gigabyte is roughly a billion bytes while a terabyte is one thousand gigabytes—the average novel is around 50 thousand words totaling only a few hundred thousand bytes total.

Add in the fact that ChatGPT stores previous input and output to create a working memory for itself to reference and contextualize future responses, and you’ve got quite a formidable machine. Having a tool that you can ask questions of in the same way you’d ask a friend is incredibly helpful for brainstorming new ideas. Not many people are able to figure out how to ask computers for things, so using a simpler, more intuitive input structure to solicit a response will result in a more appropriate output. I know many people including myself prefer asking a question directly to someone who would know the answer over trying to figure out the best search query to put into a search engine.

But Current AI Usage is a Problem

The fact that Large Language Models used by GPTs are composed entirely of previously created content, however, should give you a hint about why using AI in anything but a hobby or strictly inspirational context is extremely dangerous. The sheer quantity of sample texts provided by the LLM helps reduce the odds of it repeating some unique phrase verbatim, but it’s far from impossible for a GPT to generate its output in a way that’s identical to some of its sample texts simply because the ranking of the word order seemed most reasonable. All of that previously created content within the LLM was produced by someone, and if the content used isn’t strictly and thoroughly vetted, a GPT can quickly and easily steal someone’s work without the original creator, the GPT’s developers, or the person using the GPT ever knowing until it’s too late.

Prominent examples of this can be found in various art bots. More than one bot appears to (web archive) source their art from an unimaginably large collection of artwork posted publicly by the artist community on DeviantArt.com. The collected art is then used freely by the GPT to construct remixed art, at least according to some posts I’ve seen about those who have noticed their (rather specific) art styles reflected in some generated art. The fact that the artist is never informed that their art is being used can at the very least be considered unethical if not outright copyright infringement.

Likewise, ChatGPT itself has been known to source code snippets from the ubiquitous home of software developers, GitHub, in order to produce (often functional) code for software. Unfortunately, just because the code on GitHub is publicly visible, most projects use one of a huge number of licenses that dictate how the code may and may not be used. Unfortunately, it seems that these various licenses were completely ignored when the code samples were collected for ChatGPT, which has resulted in some hilarious outputs that include the text of the restrictive license and the name of the writer of the original code!

In addition to plagiarizing content, most of the output that GPTs produce is also factually inaccurate because, again, words are simply being combined in plausible ways based on how they have previously been used; there’s no analysis of those words nor understanding of the topics. Likewise, biases and opinions of previous writers are assimilated into the word proximity data, so the output must always be checked for accuracy, tone, and underlying assumptions, and only a person can do that.

There are infinitely many other ways that a GPT can fall short when its LLM is not carefully curated and its output is not rigorously reviewed. Since LLMs are not carefully curated in this way, it falls to the users to be extremely diligent in massaging the output into something truly unique and correct instead of just the amalgamation of billions of things produced before.

Beyond plagiarism, many employers and content producers have already attempted to cut costs (web archive) by replacing their employees (web archive) with AI powered by GPTs. This misguided overreliance on AI is a huge problem, not only for the real human experts losing their jobs but also for users of the end product, which quite possibly contains wildly inaccurate information. Search engine results were already clogged with pointless SEO articles, and now they’re being flooded with blatantly false information that has been generated by AI and has not been vetted at all. Relying on a GPT to do work that only a thinking person can do is not only unethical but also results in content that is low quality, derivative, and uninspired by the very nature of what LLMs are.

How to Use AI Properly

AI is a tool, and it must be treated as such, guided by an intelligent hand. Recall that GPT stands for “generative pre-trained transformer,” and remember that it is only transforming previously created content. Considering how GPTs function and how they construct an output, it’s clear that the most useful applications of ChatGPT and other tools like it are brainstorming and templating.

Many writers will attest to the value of talking to someone when they need help finding the right word or phrase while writing, and ChatGPT can be an effective stand-in to help conjure up the needed word combinations. Likewise, the monotony of some repetitive writing tasks can be alleviated; for example, asking ChatGPT to construct a sample thank-you note that you can fill in with accurate personal details after a job interview. One clever tool (Goblin Tools) uses ChatGPT to help expand to-do lists into smaller pieces, analyze text for tone, and perform other helpful actions specifically to help people with executive dysfunction issues or other neurodivergent traits.

The important thing is that you treat everything it produces as a starting point that you then mold into a final product. Simply using what ChatGPT outputs without making changes is only asking for trouble. This is especially true when you’re making content that people rely on to be accurate. The only exception is when using GPT output for personal entertainment—fun doesn’t need to be accurate.

Moving forward, it is critical to be conscious of what AI is for and to understand when and when not to use it. Asking an AI to generate a story about your child’s favorite characters going on an adventure will always be more appropriate than using it to write an entire novel you plan on distributing. Using ChatGPT to give you a to-do list when packing for a trip to another country will always be more ethical than using it to replace people who make content for your website. Just as GPTs rely on context to produce output, you must also understand what context AI output is useful for. When in doubt, tidy up the output anyway—the results can be good, but they are rarely great.

Humans are Not Obsolete

At the end of the day, ChatGPT and the impressive things modern AI is doing truly can’t replace you. The words they write with are only what you have written in the past, and their assertions only reflect what you have asserted before. They cobble together coherent sentences only because of how you have previously used words, and they disregard the truth because they can’t analyze what they write. A tool is only as good as the hand that wields it, and ChatGPT is only as effective as the creative mind that directs it. AI can help you find different ways to express ideas, but only you can create original and authentic content.

toki! mi jan Lapi li kama sona e toki pona!
⬇
“Hi! I’m Robbie, and I’m learning toki pona!”

toki pona is a fully usable constructed language with just 120 created words by Sonja Lang in 2001 that has since grown to 137 essential words with thousands of speakers and writers around the world! I wanted to try learning toki pona ever since I first heard about it, but I never had any particular reason to (the same reason most of my language learning endeavors have stalled in the past). But now I’ve decided to ignore my ingrained apprehensions about its practicality and just learn it for the sake of knowing it.

This is notable because I very rarely consider learning to use a conlang, including any of my own! Conlangs have always been an intellectual interest for me—I love seeing how people choose to communicate different things and what kinds of sounds they use to do it. It comes from the same roots as my interest in linguistics and how languages work and are formed even though I don’t often go out and try to learn a new language. But toki pona is special… And only 137 words.

There are now 2 books written and compiled, respectively, by Sonja Lang for learning the language, Toki Pona: The Language of Good and Toki Pona Dictionary, but there are so many excellent resources for learning the language available online that these books are essentially just insightful resources for further understanding the language. Toki Pona Dictionary was released in July 2021 and contains around 400 pages of toki pona phrases that have been used by the community and references for how various English phrases can be communicated in toki pona.

Here is an example of what toki pona looks like:

mi kama sona e toki pona la, mi kepeken e toki pona lon tenpo ale tawa ijo lili. kin la, jan ale li wile toki e toki pona tawa musi!

It says, “When I learn toki pona, I will use it all the time for little things. Indeed, everyone should speak toki pona for fun!” I’m still working on learning the grammar and vocabulary, but I’m pretty sure I got it roughly correct.

Anyway, it’s a fun language that forces you to simplify how you think in order to communicate what you want to say. There is no distinction between happy, satisfied, content, or fine—it’s all pona, “good.” There’s no distinction between sad, depressed, angry, or evil—that’s all ike, “bad.” At the same time, the words and how you combine them do a lot of interesting things together that can communicate a lot. Perhaps you can’t directly say that you’re sad, but you can say that you’re pilin ike, “feeling bad.” Suppose you’re not sad, but your stomach hurts? You could still say you’re pilin ike, or you can specify that insa mi li pilin ike, “my stomach feels bad.”

See what I mean? It’s a little puzzle to figure out how to communicate, and context is king. You need to know at least a little bit of context to know exactly what’s being communicated in many cases. That’s what I love about it though. You can’t talk about advanced programming topics or math, but you can talk about your day or things you did in a simplified way.

The Master List

When I decided I wanted to learn toki pona for real, I started watching a very good video lesson series by jan Misali on YouTube that introduced 10 words at a time with grammar rules in roughly 3-minute chunks, and it was very helpful for getting me up to speed very quickly (with lots of pausing to copy down words). I have since gotten a copy of Toki Pona: The Language of Good (called pu in toki pona), and I learned that the Toki Pona Dictionary (called ku in toki pona) was released just a couple months before I started learning! In doing research on that, I learned (from this video by the same person who did the lessons) that 17 new words were made official to the language!

Unfortunately, it’s so new that all of the dictionaries online were only of the “classic list” included in pu plus a small handful of other words that were somewhat commonly used. So I set out to compile a master list of all 137 official words that also included some helpful word combinations to help me study. Using the dicitonary in pu and 2 other websites that had expanded and altered word meanings, I compiled the ultimate toki pona word list! (Those lists are from tokipona.net and devurandom.xyz/tokipona plus this page from the same site, in addition to some definitions from jan Misali’s video on ku linked above.) Using my sources, I jammed every meaning and part of speech into the list to help make sure I knew not just the meanings of the words but also certain nuances on them.

Once it was compiled into a text document, I went through and copied all the words and definitions into the only flash card study app I could find for iOS that was fully free, flashcards.io. Lucky for you, the app is primarily web-based, so if you want to study toki pona vocabulary with all of the words, you can check out the flash card deck here:

toki pona ku (en nimi ilo) on flashcards.io

Unfortunately when I checked on the link above a month after posting this, the link no longer appears to work! So I went ahead and moved it to Anki, the most popular flash card app that I know of, which may be more helpful for you:

toki pona ku (en nimi ilo) on Ankiweb
(Note: Only the desktop and Android versions of the Anki app are free, but if you create an Ankiweb account, you can sync your progress with the desktop app and access your decks from your iPhone for free in the browser!)

I intend to expand the list as I learn more toki pona and common word combinations, but the official 137 words are in there, and they won’t change.

So go learn a new little language! o tawa kawa sona e toki pona!

Table of Contents

• Introduction
• Getting Started
• Getting Your Export
• Extracting Your Export
• Processing Your Export
• You’ve done it!
• Other Options and More

Introduction

Google is amazing, and they have provided many great services for many years. Google also has a history of shutting down projects that many people rely on with very little warning. While Google isn’t shutting down its Photos system by any means, they are ending free high quality uploads on June 1, 2021 (except for Google Pixel phones exclusively); at that time, photos uploaded to Google Photos will start counting toward your (still very generous) 15GB of free cloud storage that’s shared with Google Drive and Gmail.

If you’re like me, you’ve enjoyed Google’s free photo upload system as a wildly convenient backup platform for all the photos on your phone, but with the free version getting cut and storage space shared with Google Drive, I suddenly find myself with a very small amount of storage space remaining! First, I do need to make it clear that Google’s pricing for storage is very good at about $2 per month or $20 per year for 100 GB, and if you’re able to pay the monthly or yearly fee, Google Photos is (in my experience) unmatched in its convenience and features! Just keep it in the back of your mind that Google may shut down this system at any time in the future as well. :)

If you’re not willing to pay Google and/or you’re concerned about your data privacy (or maybe you just want to get your photos off Google to put on a different cloud hosting platform), read on! Do be warned, Google doesn’t want you to switch platforms, so they make it as inconvenient as possible to accomplish your goal. It’s annoying and it takes a long time, but you stick with it, you can do it and this guide will help you!

Getting Started

The first thing you need to do is tell Google you want to initiate a data export from Google Takeout. As far as I can tell, Google Takeout lists every Google service that stores data of some kind, but this guide only covers Google Photos.

Optional Recommendation: Before exporting your photos, take the time to go through your Google Photos collection and delete as many unwanted or duplicate photos as possible! This will make the process so much easier because you’ll have that many fewer photos to deal with down the line.

Google Takeout

When you get to the Google Takeout page, first click “Deselect all” to uncheck all the checkboxes, then scroll down to find Google Photos and check the box next to it:

Note: Using the button in the Google Photos section for “All photo albums included,” you can customize what albums you want to include in the export; the “Multiple formats” button simply explains the types of files that will be included in the export. In my case, every individual day that a photo was taken got separated into its own album.

If you’re exporting all your photos, this doesn’t really matter, but just be aware that choosing specific albums can take quite a bit of time…

After checking the box, scroll all the way down and click the “Next step” button near the bottom of the page. This will take you to the “Choose file type, frequency & destination section” where you have a few options.

Export Settings

The first option is “Delivery method,” which lets you choose whether to have Google email you links to your files when you download them or to have Google directly upload the messy Takeout folders directly to a different cloud storage service. This guide will be covering the Email method so you can get the files directly onto your computer and subsequently organize them into usable folders.

The next option is “Frequency,” which is useful if you plan to continue using Google Photos, have a ton of storage space for backing everything up multiple times, and want to perform this process multiple. Otherwise, just choose “Export once” and move on.

Last in this section is the “File type & size” section. If you’re not technically inclined, don’t have any long videos stored, and don’t want to think too hard about things, just use the default selections of “.zip” and “2 GB” and click the “Create export” button—we’ll meet back up in a couple paragraphs. If you want to explore these options (or do have large video files), read on! The file type options are “.zip” (the format just about every computer can handle) and “.tgz” (the format that results in smaller file sizes but takes more effort to extract). If you’re just exporting a backup for storage and do not need to look at your pictures, go ahead and use that .tgz format—it should provide more efficient storage on your external drive. Otherwise, you should use .zip for the convenience.

The file size option is an interesting one with a very important and unstated consequence for choosing a size that’s too small. If you’re using Google Photos like me, then you probably have a lot of photos stored on there (or example, I had about 61 GB of photos and videos)! This option lets you choose how you want Google to package up your photos—your export will result in however many .zip (or tgz) files it takes to chunk up your photos into your selected file size. I originally chose the default 2 GB, so my 61 GB of files resulted in 30 .zip files that were 2 GB each and 1 file that was about 128 MB or something. This is an interesting choice because this means that you will need to download that many files to your computer and manually unzip them all! The tradeoff here is faster download times versus longer processing time, where smaller .zip size will result in faster downloads but more processing and vice-versa!

Once you’ve made your choices, click “Create export” to move on.

Getting your Export

Google will begin processing your export, and this can take anywhere from 10 minutes to an hour to a day depending on how many photos you have! My export only took about 20 minutes, but allow more time if Google’s servers are busy or you’re a professional photographer with hundreds of thousands of photos. Either way, you’ll get one email telling you that the export has started and another telling you when it’s ready to download!

Once you get your email telling you your export is ready (or if you stay on the page until it’s done), you’ll get a link to the “Manage your exports” page. Unless you’ve done an export before, it should only show one item with a clear “Download” button. When you click it, Google may ask you to re-enter your password to log in. It will do this if you don’t complete your work within a comparably small amount of time (maybe every 5 or 10 minutes?), so keep your password handy if your browser doesn’t fill it in for you automatically!

Click that download button, and depending on the size of your export, you’ll get a popup on the screen with a list of files to download. Just download each of them one by one—every time you click the download for one of the items, it will refresh the page and initiate the download. Just confirm the download if your browser asks for confirmation and do this for each file in the list.

Each time the page reloads, the icons inside the buttons next to previously downloaded items will change from a download icon to a checkmark icon.

For most people, this download process will be long and tedious. Just stick with it and keep downloading until you’ve got all of them. You need every single one of the downloads because your photos and the associated files are just randomly scattered among them. This is one of the things I was talking about when I said Google made it as inconvenient as possible.

Extracting Your Export

Now that you’ve got all of your export files, you get to begin the “fun” process of extracting and merging them all into a single folder! This process is different for MacOS than it is for Windows, but I’ve got you covered. You don’t need any extra software if you’re using .zip files on MacOS or Windows, but I do recommend you use the most recent version of 7-Zip on Windows (or p7Zip for Linux)! 7-Zip is free (and safe!), and it can handle an outrageously wide variety of different compression formats!

Like I mentioned before, it is important that you have all of the files from Google and that the downloads are all complete, because you need to extract the contents of these files and merge them together. Your operating system should handle the merging process for you, but this guide will show you how to do that.

On Windows

All of my screenshots will be from Windows 10, but it should be a similar process for older versions (XP, Vista, 7, or 8) when using 7-Zip, linked above. (The process should be similar for Linux too, as long as you’re using a graphical UI like Unity in Ubuntu.)

Ok, this process is not difficult, it’s just mind-numbing and tedious—just pay attention and try to make sure you click and drag as precisely as you can with your mouse. All you need to do is open the 7-Zip File Manager and navigate to where you downloaded the .zip files from Google and double-click the first file. 7-Zip will open the file and show the contents—the first folder will be a folder named “Takeout”. If you open that folder, you’ll find a Google Photos folder, and then within that are folders with a random set of your albums, but you shouldn’t need to worry about that!

In a File Explorer window, open up or create an empty folder and put that window side-by-side with 7-Zip:

Then click the Takeout folder in 7-Zip to highlight it and then click and drag the Takeout folder into the empty folder in the File Explorer window! 7-Zip will first extract the Takeout folder, then Windows will copy the contents into your empty folder.

And that’s what we’re going to be doing with every single file you downloaded from Google until you’ve done it with all of them!

Every time you repeat this click and drag process, Windows will start asking if you want to merge folders. Answer Yes, and when possible check the box that says “Do this for all current items” to prevent Windows from asking you multiple times.

If you accidentally extract the same folders twice, Windows may warn you that there are files with the same names and ask if you want to “Replace the files in the destination” or “Skip these files”. Typically this only comes up when you’ve already extracted the folder before and you’ve accidentally initiated the copy again, so it is safe to choose “Skip these files” to shorten the time it takes to extract. But if you don’t trust yourself, choosing “Replace the files in the destination” won’t hurt anything, it’ll just lengthen the time it takes to finish extracting a little.

On Linux

The process on a GUI Linux using p7Zip will be nearly identical, with the main difference being the way your operating system asks you about merging the files. The process of clicking and dragging from p7Zip into a folder should be exactly the same, though!

On MacOS

The process is a bit different in MacOS because of how MacOS opens extracts things from .zip files: instead of opening in a new window where you can preview the contents first, it just extracts the files immediately into the same folder! So after you download your export files, I recommend you create a new, empty folder and move all of the .zip files into that new folder.

Once you navigate to the folder with all your .zip files inside, simply double-click each file, and its contents will be extracted for you! It may take a while to extract the large exported files from Google, but just be patient and let the computer do its thing.

Next, you’re going to merge each of the “Google Photos” folders into the first “Takeout” folder. Pay attention, because merging folders in MacOS is not as straightforward as in Windows!

To enable the option for merging folders, you need to perform a Copy of the folder in question. You can do this in one of two ways:

1. Select the folder, use the Copy function, and Paste it into the correct folder (more steps)
2. Hold down the alt option key as you click and drag your folders into the target folder (one step)

Copy/Merge Method 1

Method 1 (using Copy and Paste) might be more familiar to most people, but you have to make sure you’re copying the right folder and pasting it into the correct place or else you’ll just be duplicating many gigabytes of data onto your hard drive.

There are at least 3 different ways to access the Copy and Paste function on MacOS:

1. Select the folder you want to copy, click Edit at the top of the screen, choose the Copy "Folder Name" option, select the folder you want to paste into, click Edit at the top of the screen again, and choose the Paste Item option.
2. Select the folder you want to copy, hold ⌘ command, and then press C to copy, open the folder you want to paste into, hold ⌘ command, and press V to paste.
3. Right click the folder you want to copy, choose the Copy "Folder Name" option, then right click the folder you want to paste into and choose the Paste Item option.

To use this method, open the “Takeout 2” folder, copy the “Google Photos” folder, navigate to the first “Takeout” folder, and paste into that folder. The computer will prompt you whether you want to cancel, replace, or merge the folders—choose “Merge” and the computer will handle the rest!

Just make sure you are copying and pasting into the correct folder and getting the “Merge” option. If you are pasting into the wrong place, you might accidentally use up a lot of space on your hard drive by accident! To help manage your computer’s storage space (and to help you remember which folders you’ve already merged), go ahead and delete the Takeout folder you just merged from after the merge is complete!

Copy/Merge Method 2

Method 2 (using alt option with a click and drag) takes a little bit more precision, but in my opinion is a lot easier to do right if you’re careful!

So expand the “Takeout 2” folder so you can see the files in one window, highlight the “Google Photos” folder within, hold down the alt option key on your keyboard (it’s the button next to the ⌘ command key on your Apple keyboard, in case you’ve never noticed it), and click and drag the “Google Photos” up until it highlights the first “Takeout” folder. Then just release your click! The computer will prompt you whether you want to cancel, replace, or merge the folders—choose “Merge” and the computer will handle the rest!

Hold the `alt option` key while clicking and dragging

Repeat this merge process with each extracted Takeout folder, remembering always to hold down the alt option key while clicking and dragging! If you ever get a prompt from your computer that does not include a “Merge” option, cancel and try again! If you’re sure you’re doing it right, you may have already merged that folder in. Feel free to cancel or replace if you don’t trust that the merge happened.

To help you remember which folders you’ve already merged (and to help manage your computer’s storage space), go ahead and delete the Takeout folder you just merged from after the merge is complete!

I can’t stress enough the importance of getting your aim right when doing this. Make sure you are dragging the “Google Photos” folder from each Takeout folder into the first “Takeout” folder! The Takeout folder will highlight when you are hovering over it correctly. If you accidentally drop the folder into the wrong place, it will create a copy of the folder in that other folder!

Another note, the view I’m using in the example is “as List”. If you’re not seeing the files in your folder on individual lines like in the screenshots, make sure the Finder window with your files is selected, click “View” in the top bar, and choose “as List” from the menu.

Processing Your Export

When you’re finally done extracting and merging, you’ll have all of your images from Google Photos in one place on your own computer! You did it!

But if you actually want to be able to look through your pictures in any reasonable way, you’ll need to do something about the bonkers folder structure that Google left you with. To do that, you would need to go through and organize things yourself, which is commendable and perfectly respectable, but it’s a huge pain in the butt and takes absolutely forever.

Luckily, I made a script for myself during the process, and I’ve tweaked it and made it into a user-friendly little program for others partaking in this unfortunate journey after me!

The program goes through your extracted Google Photos folder and moves every image and video it finds into nicely-organized folders based on the year and month the photo was taken (or created)! It can also extract and merge the files for you to save you the effort of doing that manually!

Get the Google Photos Export Organizer

On the page linked in the button above, download the file labeled for your operating system (Window, MacOS, or Linux), extract the folder, and run the program inside to get started.

It will take you through a series of 4 steps to help you get your Google Photos export taken care of. Step 1 points you back to this post for how to export and download your files, then Step 2 will extract your exported files for you! Just tell the program where you downloaded your .zip files and where you want to extract them, and it will do it for you! Step 3 will move the files from your extracted folders into new folders organized by year and month that the photo was created. Finally, Step 4 reminds you to delete the exported files to help you clean up.

I think it’s fairly straightforward and easy to follow, but if you run into issues or have any questions, please submit an issue on GitHub so I can work on fixing it as soon as possible! If any fixes need to be made, the program will alert you about any available updates to download when it loads.

After running the program on your folder…

You’ve done it!

You can now do whatever you want with your nicely organized photos! You can upload it to a new cloud platform, you can keep it on your computer… Whatever you want! Your photos and videos are just plain computer files, and you can do whatever you want with them.

Don’t forget to delete all those .zip files if you don’t need them anymore! They probably take up an awful lot of space on your computer.

Just one last reminder: all of the files you just downloaded and processed are still on Google’s servers, so if you want to completely cut the Google cord, you’ll have to delete them from there.

Enjoy being disconnected from Google Photos!

Other Options and More

Since posting this, I’ve seen a lot of great discussion and suggestions around this subject. I’ll use this section to add more tips and options for exporting and organizing your Google Photos that might be more effective depending on your specific needs as I am able to compile them!

In the meantime, check out the comments below and the comments where I posted on Reddit for some good discussions about alternative options to Google Photos and potential issues with using the Google Takeout system.

I knew from the start that such a project would be incredibly ambitious and take a lot of work, but I was sure that if I didn’t start working on it, I would never see the alternative that I wanted so badly (thankfully other projects have begun since then). I’ve been a part of the Fediverse for a few years now, so I wanted to build an application that could interact with other apps via the ActivityPub protocol. I also am a huge fan of making software that’s easy to maintain and use, so I wanted the app I built to be as easy as possible for a tech-minded person to contribute code to and run their own instance of. These things all resulted in me leaning hard into setting up the Readlebee project as a “real deal” Open Source project that used frameworks that are relatively easy to learn that could be run using just NodeJS and the user’s choice of database manager (as opposed to requiring a suite of specific software). I looked into community guidelines, researched the appropriate software license, and set up the repository on the best platform for easy contribution (GitLab, because allows logging in with a good number of popular social media accounts).

My goal was to do things “right” from the start instead of my normal way of just getting it done quickly and going back to clean things up when it was working (if I got to it). My main reason for doing this is because this project is supposed to be open source and therefore easy for others to read and contribute to (instead of my normal personal project that’s easy for just me to follow). While I do believe that this approach is a good thing, it is the biggest reason that my progress has been so slow. Well, that and the fact that I ended up reacting unexpectedly poorly to the COVID-19 pandemic—that had me out of commission in a negative spiral for some time at the beginning of 2020—but I digress. Doing things the “right” way meant using code frameworks that I was not familiar with, so I had to learn new systems while also building my own. I’m also trying to ensure that Readlebee has multi-language support and the best accessibility I can provide from the get-go, which has further slowed things down. All this extra learning on top of occasional hangups (plus actual full-time work) has led to an extremely slow development cycle without a lot to show for it after a year, and that’s super discouraging to me.

I’ve been doing my best to recognize that this lack of visible progress is okay because I’ve been doing a lot of learning and planning in addition to writing code for the project. Plus trying to account for doing things “right” the first time should mean less going back to fix things in the future, which will theoretically save time later. It’s just a very different development process than I’m used to, so it feels wrong to not have more done.

I’m hoping that the coming year will see at least a beta release of Readlebee so I can start getting some help from other developers plus feedback and ideas everyday users. I’m itching to have something to show for all my efforts, but no matter how hard I try, I can’t seem to rush this project. But maybe that’s a good thing in the end.

If you’d like to keep track of Readlebee’s development process, feel free to star the code repository on GitLab or GitHub (mirror) to see how you can contribute and follow my Readlebee Mastodon account to see occasional updates about my progress on the project as well as other FLOSS- and tech-related topics!

Today, I decided to end development on a board game I’ve been working on for about a year. It’s gone through many iterations, and even though I think the idea might be interesting, I don’t believe that the game can ever be enjoyable. While I usually think that’s totally fine to make games just because the idea is interesting (I’ve released my fair share of “interesting” game ideas), I feel like this case is different because I intended for it to be an enjoyable. As a general rule, I make games that I think I will enjoy, and after a lot of thought, I realized that even I wouldn’t enjoy my board game.

After making the decision, I’ve been looking back through some of my old game ideas, realizing that a lot of the ideas I love are far outside of my skillset to create effectively. The thought of simply throwing the ideas away is too painful, since I’ve already written so much about them, but I’ll probably never do anything with them. This dichotomy let me to the thought of publishing my game ideas—if I’m not gonna be able to do anything with my ideas, maybe someone else will! If I publish them under a Creative Commons license, then maybe I can work with someone who might want to use the ideas.

I’m under no illusions that anyone would ever want to use my ideas, let alone that anyone might actually ever discover them, but the thought of simply letting the ideas rot in Google Drive until the service goes offline is just as absurd. If my ideas could even have the potential of being used, then that would be very exciting to me. (It would also be nice to prove to myself that I’m not just sitting around doing nothing—I can look at my ideas cache and know that I’m at least thinking of things.)

So very soon, I’m going to be updating this website to make viewing my game ideas possible. I’m not sure exactly how I’ll style it yet, but as long as I have them visible, I think I’ll be happier.

I hope you enjoy the ideas as much as I do!

For regular static HTML sites or apps that only use JavaScript, it’s hard to go wrong with GitHub Pages. By default, GitHub Pages allows you to host either raw/pre-compiled HTML/CSS/JavaScript or a Jekyll site, and this website is using GitHub Pages with a Jekyll setup—you can see the source code here, if you’d like! The one thing you need to be aware of is that only public code repositories are free on GitHub (actually, as of 2019 this isn’t true anymore!), which means that you have to be OK with the source code of anything you host being 100% visible and findable.

If you’d like both a less restrictive hosting setup than just Jekyll and access to free private repositories, GitLab Pages is a great option! GitLab pages are managed via their continuous integration (CI) service, which means that you can set up the CI loader to load any static site generator to process your code. For example, my GUTS+ website is set up to use Hugo by using this config to download the target version of the Hugo binary straight from GitHub:

image: ubuntu

before_script:
# Download Hugo 0.37.1 release
# Reminder: Get the release asset id from https://api.github.com/repos/gohugoio/hugo/releases if you change versions
- apt-get update
- apt-get install -y curl git
- "curl -LJO -H 'Accept: application/octet-stream' 'https://api.github.com/repos/gohugoio/hugo/releases/assets/6437077'"
- tar -xzf hugo_0.37.1_Linux-64bit.tar.gz

pages:
  stage: deploy
  script:
  - ./hugo --config config.yaml
  artifacts:
    paths:
    - public
  only:
  - master

Or alternatively, this story I’m attempting to write uses a custom-built PHP static site generator that I made because I just needed something quick and easy to throw markdown text into a template:

image: php:7.1.1

pages:
  stage: deploy
  script:
  - php ./site-generator/generate.php
  artifacts:
    paths:
    - public
  only:
  - master

Both GitHub and GitLab Pages allow you to add custom domain names to your hosted site, which can get a little tricky, but it’s well worth it! You can even use SSL For Free to generate Let’s Encrypt SSL for your domains to you can get that coveted https before your domain name.

If you want something really advanced, Heroku offers free access to full server instances called “Dynos”. You need their Heroku command line app in order to interact with the service, but once you’re set up and you’ve followed a couple tutorials, you can get a full website or API or whatever you need going with just your code and a little bit of config setup! This is what I’m using to control the backend API for my game Radio Silence along with a couple of other things.

GitHub and GitLab Pages are great options, but it can be a little hard if you don’t already use Git or don’t want to take the time to figure out how to set up GitLab Pages the way you want. If you want something a little more in the vein of traditional web hosting, I found a couple of services that offer 100% free web hosting with PHP and MySQL available with the only catch being a few restrictions and potentially slower websites. And also they’re generally a little bit shady…

Maybe I’ll finish this post someday. But that day is not today, so here’s what I’ve got so far.

Flash forward to a few days ago—every new feature I add is a huge victory because I have to make so many file changes and test updates just to get one small thing to display correctly, and it’s generally exhausting and not nice. They had so many checks and balances that are great for maybe a 10+ person team but that simply doesn’t work in our 2-person environment. The code is so over-egineered that nothing is easy to change, and that’s not a good thing for our company because sudden small changes are frequent and numerous.

My coworker and I were talking in whispers about how we wish we could just start from scratch to make something simpler. We had hired these other developers because I hadn’t been hired yet and they needed the extra hands, which more or less landed them in the same position as the previous Java cart—someone else wrote it so we had to depend on soeone else. The logic was that Javascript was easy enough to work with, so we wouldn’t be as tethered down! So we ended up landing on “no, we’ve come this far, we’ve just got to power through it” so we could just go the last bit of the way and get it ready for launch.

But that’s just not realistic.

That night, I started researching and found that not only was the cart not finished, the style framework they had chosen doesn’t even support the old browser versions we wanted to support! I did some digging and found that my favorite style framework, Bulma, ended up being more compatible if I used a specific version of this Flexbox polyfill. So the next day, I started secretly building the cart front end from scratch using the nice, simple choo framework and Bulma. After a day, I had already completed the basic features of adding items to the cart from our API and displaying them.

After a long meeting about the cart’s status yesterday, my coworker comes out and asks if I wanted to simplify the over-engineered code of the cart at all, and I said yes, of course. “Do you want to just try ripping out the complex stuff or just start from scratch”. I said I thought it might be easier to start from scratch and go from there, so he told me that makes sense and that we could just try going in that direction. At which point I let out a sigh of relief and showed him my 2-day progress.

I know it’s not going to be easy starting over, but at least I’ll be a lot happier working with it.

Yesterday, I spent two hours trying to solve a problem I was having with an API I had built to store our products in a unified place. Every single product that I had ever tried to grab from the API had succeeded, but as I was going through the database to update some values, I came across one product that would not load. No matter what I tried, no data was returned and no errors were logged. I was baffled.

I went through the code and was able to narrow it down to one problem method in the API’s class, but when I tried to run that same method in isolation on the product that was failing, it worked fine! After trying to narrow it down even more, I finally decided to try looking at the contents of the product. In our product database, we have both regular products and packages, which can contain any number of other products and/or packages. So I checked every sub-package in the package I was having trouble with, and sure enough, one of the packages inside was also not loading.

I kept digging, and about 5 packages deep, I found the product that was the culprit. When I looked at the product’s data in the MySQL database, I discovered this: Gabriel García Márquez. I figured that for some reason, these í and á Unicode characters had to be the problem. So I replaced them with their non-accented forms, and just as I suspected, every package loaded just fine. For some reason, my PDO connection was unable to comprehend the Unicode that I know actually does work just fine.

Obviously, however, I don’t want to have the data be wrong just because some programming language can’t handle it, so I went ahead and updated the database to contain the accented characters using PDO. Once I did it that way, it was able to load the Unicode without any trouble! But when I looked in the MySQL table again, I discovered this mostrosity: Gabriel García Márquez.

WHAT!?

Our server, MySQL database and tables, and PHP config were all set to use the utf-8 charset to support Unicode characters, but for some reason, the PDO library is translating the Unicode characters into other Unicode characters in order to store them and re-translating them again when it pulls the data!

WHY

So I went on a Google hunt.

After finding a lot of issues where people had their charsets set to latin or some other non-unicode set, I finally discovered this StackOverflow article where someone was having a similar problem. The marked answer from 2011 includes an apparent workaround that includes PDO connection options inside the connection call:

$pdo = new PDO( 
    'mysql:host=hostname;dbname=defaultDbName', 
    'username', 
    'password', 
    array(PDO::MYSQL_ATTR_INIT_COMMAND => "SET NAMES utf8") 
); 

But the next answer down from Palec, with only 31 upvotes, contains the actual, correct answer—or at least, the correct answer… since PHP 5.3.6:

You have to set correct charset for the connection. Add charset=utf8 option to the DSN. (This is MySQL-specific!)

(Emphasis added)

When going to his link to the PHP documentation, I discovered this:

charset

The character set. See the character set concepts documentation for more information.

Prior to PHP 5.3.6, this element was silently ignored.

WHYYYY

When looking at the regular PDO connections page, where every Google search for “PHP PDO Connection” points you, you see nothing but this example, which doesn’t even reveal the fact that you can add PDO options to the connection:

$dbh = new PDO('mysql:host=localhost;dbname=test', $user, $pass);

But lo and behold, searching the page for “charset” yields a single comment at the very bottom of the page, downvoted to -3, is the correct answer about specifying the charset option in the DSN. Why is it downvoted? Why is it so hard to find this charset information? WHY does the charset need to be specified in the connection when EVERYTHING leading up to it has already specified the charset to use??

I have no idea.

In my opinion, the connection should default to the same charset of the database that is being connected to, because that just makes sense.

Anyway, I just want the world to know that the correct way to connect to a MySQL database with any PHP version since 5.3.6 is:

TL;DR

If you want to have a clean database with correct Unicode characters, always be sure to specify the charset you wish to connect with. You can do this with PHP’s PDO as follows:

$pdo = new PDO(
    'mysql:host=hostname;dbname=database;charset=utf8',
    'username',
    'password'
);

I hope this helps someone the way it surely would have helped me!

It’s been difficult for us to work with them, though, because they have a larger team than us, and thus use a more structured approach to development, namely Agile. Since my coworker and I are just a two-man team, we don’t have a structure like that because we can simply work to get things done quickly whenever we want. This lack of structure and order has clearly been irritating to the other dev team, but Agile would only slow us down.

Fortunately it’s not too much of a problem, but it has led to some holdups. Earlier in the development timeline, they were waiting on us to deliver an API for our until-recently nonexistent Product Database so they could pull product data, but we were trying to figure out a sustainable way to build product packages in our database so they could be maintained better than they currently were. It took a long time, but we finally got the system up and running for them, and they were able to resume development. The frustrating thing was that it felt like they weren’t doing other things that clearly needed doing that could be done without the product database being finished.

Unfortunately, that attitude and feeling is mainly because we don’t use their development structure—for them, it takes a lot of extra planning and work to even prepare to do things where my coworker and I would be able to simply start working on things and figure it out as we went! We’re on a speedboat while they’re on a battleship, and there are good things about each. It’s been difficult and tricky, but we’ve finally found a way to work together. They’re able to work on other things while we prepare backend and API tools for them to use when they reach where we are. Our speed allows us to prepare for their arrival, and so far it seems to be working out nicely!

I just hope that it’s enough to get them to where we need them to be by October when we finish working with them and take over their project ourselves… Time will tell, I guess.

Today, a bunch of our friends helped us move out of our third-floor apartment, and I can’t express how helpful it was. It saved us a boatload of money and it made the whole process go incredibly fast! We had everything out of the apartment and into the house in about 4 or 5 hours! Amazing! We got a 16-foot moving truck from Budget, and we were able to fit everything in that truck and some of the more delicate stuff into a few of our friends’ cars. It ended up being a lot of fun, too, because we all hung out afterward and ate pizza and chatted.

Some friends stayed even later and went to dinner with us! It was amazing that everything went off without a hitch, and I am so glad.