I feel like you can only take this logic so far though, right?

The examples you've used to demonstrate where the "Bible says so" mentality is bad (slavery, apartheid, etc.) make sense. But those are things that society has opinions on independent of the Bible.

But what about ideas that only exist in the Bible? For instance, that Jesus is God's only son, and died for our sins? Is it acceptable to quote the Bible when making strictly theological claims? Or where do you draw the line?