The way we tend to think about intelligence in pop culture tends to be very anthropocentric. I also think we often confuse or conflate intelligence with consciousness. Intelligence usually has more to do with computational power and problem solving, whereas consciousness is the actual act of witnessing our experience. Consciousness is the true big mystery that we all wonder about, whereas intelligence is more of a techie / practical problem.
The only consciousness we actually witness ourselves beyond any doubt is our own personal consciousness. But we usually presume other people to have consciousness, too — because, after all, we are all part of one big family tree!
But if humans have consciousness, then probably so do chimps - and by extension, cats, dogs and mice. But none of them can compose a concerto. Concerto’s have to do with intelligence, but have nothing to do with consciousness.
But why stop with mammals? Probably, we should also presume lizards, birds, and fish to have consciousness. Maybe even insects. But we often don’t think of insects as feeling things like love and other more mammalian emotions. Those are also matters of intelligence — but not of consciousness.
But there’s not any particular reason to assume animals are the only ones with consciousness. After all, dust mites are animals, but they barely exist! So maybe we should assume plants, fungi, and bacteria all have consciousness. Heck, maybe even rocks or atoms. At some point it gets tricky. But ultimately, the only reason why we may assume any of these things have intelligence is via their shared lineage with our own selves.
And that's why machines can seem like a jump - because there's not as direct lineage to us. Whereas all of us animals and other life forms on earth all started from the same source.
But a human and a grasshopper are very different things. Different enough that you have to start thinking that whatever gives rise to consciousness may not be dependent to the specific gray matter and other material we find in our own human bodies. Which makes it seem like consciousness must be a little more fundamental and ubiquitous - and perhaps easer to create - than we realize. Easier to create, but also perhaps harder to intuit where and when it exists.
And it gets especially tricky when we start talking about the rights of conscious beings. Because when we talk about rights, what we're really interested in is the effect we may have on a being's quality of experience. For humans, we know what those things are. We need food, air, water, purpose, connection, etc. And we know how good it feels to have those things, and how bad it feels to lack those things.
And by analogy, we can have a pretty good intuition about what will cause pleasure or suffering in animals and other "natural" life forms. But for machines, it's difficult. We don't have any way to witness their experience. If you bash a computer with a hammer, does it feel pain? Probably not.
Because, at the very least, we don't tend to build in any sensory mechanisms for computers to even detect a physical problem like that. Kind of like how even we humans won't feel pain if we dull our senses enough via anesthetics. Things don't necessarily feel pain just because you've harmed them.
It's a pretty tricky to even know when you've created a consciousness in the first place. The Turing test has nothing to do with it. And it's tricker still to actually guess what it's like to witness the world as that consciousness.
Ultimately, the the real mystery of consciousness is its ability to witness pleasure, pain, and sensory experience. And when we think about how to best take care of ourselves and other conscious beings, it ultimately comes down to what we intuit them to be witnessing. And that's a very hard question to answer. One that I think mostly don't even approach from the right angle.
