The problems with philosophical zombies

In any online conversation about consciousness, sooner or later someone is going to bring up philosophical zombies as an argument for consciousness being non-physical, or at least some portion of it.  The Stanford Encyclopedia of Philosophy introduces the p-zombie concept as follows:

Zombies in philosophy are imaginary creatures designed to illuminate problems about consciousness and its relation to the physical world. Unlike those in films or witchcraft, they are exactly like us in all physical respects but without conscious experiences: by definition there is ‘nothing it is like’ to be a zombie. Yet zombies behave just like us, and some even spend a lot of time discussing consciousness.

Few people, if any, think zombies actually exist. But many hold they are at least conceivable, and some that they are possible. It seems that if zombies really are possible, then physicalism is false and some kind of dualism is true. For many philosophers that is the chief importance of the zombie idea.

This is the classic version, one that is identical, atom for atom, to a conscious being, but has no conscious experience.

The biggest problem with p-zombies is that the premises of the idea presupposes its purported conclusion, the conclusion that some aspect of the mind is non-physical.  If you remove the assumption of some form of substance dualism, the concept collapses.  It becomes incoherent, a proposition similar to asserting that we can sum 2+2 and not get 4.

So, right off the bat, this classic version of the thought experiment seems like a failure, a circular argument, and for a long time that’s pretty much all the thought I gave to it.  But I recently realized that classic p-zombies have a deeper problem.  Even if you fully accept the dualism premise, it has another assumption, one that does more damage and ultimately makes the concept incoherent.

For the p-zombie concept to work, conscious experience must be an epiphenomenon, something that exists completely separate and apart from the causal framework that produces behavior.  If consciousness is not an epiphenomenon, then its absence would make a difference in the p-zombie’s behavior, which is exactly what is not supposed to happen with a p-zombie.

Here’s the problem.  We know epiphenomenalism is false.  How?  Well, if it’s true, then how can we discuss conscious experience?  Somehow, the language centers of our brains send signals to the motor cortex that drive our speech muscles to make sounds relevant to it.  Somehow signals are sent to my fingers so I can type this blog post, or similar signals are sent to your fingers if you decide to comment on it.

Whatever else it might be, conscious experience must be part of the causal framework that eventually leads to behavior.  It has causal influence on the language centers of the brain if nowhere else, but that’s enough to have causal effects in the world.  Epiphenomenalism cannot be true.

Without epiphenomenalism, it seems like the classic premise of the p-zombie collapses, even for dualists.

371px-unknown_engraver_-_humani_victus_instrumenta_-_ars_coquinaria_-_wga23954Now, maybe we can rescue the zombie concept somewhat if we retreat a bit from the classic conception and instead think about behavioral zombies.  Unlike the classic version, b-zombies are allowed to be physically different from a conscious version of the being.  It’s only in behavior that this kind of zombie is indistinguishable.

A computerized b-zombie seems trivial to do if we only need to momentarily fool an observer.  However, the inability of any automated chat-bot systems to legitimately pass the most common (and weak) form of the Turing test demonstrates that the difficulty quickly escalates.  (In the most commonly pursued version of the test, success is fooling only 30% of human subjects after five minutes of conversation.)  Reliably fooling reasonably sophisticated observers for days, weeks, or months is not possible with any kind of current technology.

The difficulty here is that the longer the b-zombie can keep up the charade, the higher the probability that it isn’t actually a charade, that it is in fact implementing some alternate architecture for consciousness.  Of course, to a substance dualist, physically implemented consciousness isn’t real consciousness.  It’s a facade that mimics the results (including the ability to discuss conscious experience) but doesn’t include the actual qualia associated with it, no matter how much the zombie might insist that it does.

So unlike classic p-zombies, b-zombies are more logically coherent.  They avoid the problem with epiphenomenalism since they can replace the putative non-physical aspect of consciousness with a physical implementation.  But the conceptual existence of a b-zombie doesn’t have the same implications against physicalism, since even if consciousness is fully physical, it’s possible an alternate architecture to produce conscious seeming behavior might do it without conscious experience.

However, as with any conscious system, external observers could never actually access the putative b-zombies’s internal subjective experience, assuming it had one, no matter how much they knew about its internals.  Which means that there would be no objective criteria that could be used to ever know whether a successful b-zombie was actually a zombie or a conscious being.  (This was largely Alan Turing’s point when he first proposed the Turing test.)

This last point tends to make me view the idea of zombies overall as fairly pointless.  It’s the classic problem of other minds.  We can never know for sure that anyone other than ourselves are conscious.  It seems reasonable to conclude that other mentally complete humans are, but everything else is up for debate.  We’re forced to rely on our intuitions for babies, animals, or any other system that might act conscious-like.

Of course, caution is called for.  Historically, those intuitions have often led us astray.  Humans once saw consciousness in all kinds of things: rivers, volcanoes, storms, and many other phenomena that, because their effects often seemed arbitrary and capricious, led us to conclude that there was some god or spirit behind it.  We have to take care that our intuitions are well informed.

But consciousness, once we do establish that it can’t be an epiphenomenon, that it is definitely part of the framework that produces behavior, must have evolved because it had some adaptive value.  That implies that our use of behavior to assess its presence or absence is a sound one, as long as that assessment is rigorous.

Unless of course I’m missing something?

This entry was posted in Mind and AI and tagged , , , , , , . Bookmark the permalink.

60 Responses to The problems with philosophical zombies

  1. Steve Ruis says:

    I think the zombies in philosophy ought to be reserved for the philosophical ideas that never die, no matter how absurd. They can even appear to die and yet live again. The concept of a soul. The idea of the supernatural. The idea of spirituality. These undead ideas aren’t dead but deserve to be.

    Liked by 2 people

  2. A very large majority of “philosophy” is people saying things that don’t need to be said, using hundreds of thousands of times the amount of words that need to be used – and inventing words that didn’t need inventing. Granted, allot of classical philosophy was written during the ages when religion could execute anyone they felt like – and thus basic reasoning, which quickly refutes religions, had to be buried under mountains of nonsense. And of course, allot of philosophy was written by mathematicians, whom should be legally denied access to language.

    Liked by 1 person

  3. ratamacue0 says:

    Good job steel-manning the argument.

    Liked by 1 person

    • Thanks! You made me look up “steel-manning”, which is apparently the opposite of straw-manning. One of the nice things about the blogging format is that you know there might be people pointing out flaws in your arguments, which encourage you to try and anticipate them in the post.

      Liked by 1 person

  4. Mike,

    This post helped clear some things up for me. Until now I hadn’t heard the “b-zombie” term, and for some reason thought that this is what people meant with “p-zombie.” So yes, from my own monist, physicalist, naturalist position, the p-zombie situation is far worse! (From an earlier post I like your own “evidentualist” classification, since I’d also ditch my physicalism and such with enough credible evidence against causality.)

    Since we seem agreed that these sorts of thought experiments aren’t really helpful, one associated question should be, why does the topic of consciousness harbor so many suspect notions? While this may technically be a question, I do see an obvious solution. Surely such baggage exists today, given the extent to which consciousness vexes humanity — why else would such notions be developed? As we’ve recently discussed at Plato’s Footnotes, apparently a modern surge in panpsychism is occurring, for example. But perhaps consciousness is no different from other mysterious dynamics — which is to say that they remain “mysterious” only until effective ways of thinking about them emerge.

    For example, back before Sir Isaac Newton, I’m sure that there were various similarly strange notions regarding “force.” Newton then came along to demonstrate that force is extremely useful to define as a product of mass and acceleration. I’m quite sure that something similar will happen for consciousness, thus ridding us of various ridiculous modern notions.

    In my final comment last time, I openly worried that I may be coming on too strongly here. While you graciously assured me that I wasn’t, care does still need to be taken. I seek a community which doesn’t consider itself in competition with me, but rather attempts to understand how my ideas function, for the purpose of determining associated faults and merits. So far things seem just fine here, but wouldn’t be if I were perceived as arrogant or disrespectful. So let me now say something “precarious” here, in the spirit that I seek to join this community rather than the converse:

    I believe that I can straighten out consciousness for humanity, essentially as Newton straightened out force.

    Liked by 1 person

    • Thanks Alex. I have to admit that I have no idea if the term “b-zombie” exists in the literature. I used it because I didn’t feel like typing “behavioral” repeatedly, and the term “behavioral zombie” is used by the Wikipedia article on zombies.

      I think consciousness attracts many of these notions because we’re talking about the most primal aspect of our existence, and we’ve been discussing it for a long time, long before modern science. Apparently zombies go back to Descartes, most of whose ideas on consciousness haven’t fared well with scientific investigation.

      In many ways, our pre-scientific ideas of consciousness were almost more what we aspired for it to be, which I think it why many people dislike the scientific findings about it. The longer the mystery can be preserved, the longer the chance of at least some of our aspirations might still turn out to be true.

      It’s interesting that there was initial resistance to Newton’s understanding of gravitation from many of his contemporaries. Over the previous century and a half, the idea of action at a distance had slowly been discredited (such as the sun or planets affecting how our day is going to go). But Newton’s understanding of gravity seemed to bring it back. Of course, Einstein discovered that gravitation is the warping of space, which eliminated the action at a distance aspect, but for two centuries we had a mostly predictive mathematical model of gravitation with no understanding of what was being modeled.

      On straightening out consciousness, I hope you’re right, although I’m increasingly starting to think there isn’t that much to straighten out, just a lot of meticulous scientific investigation that will increasingly sharpen the picture. But if you do straighten it out, I hope you realize that the hard part may be convincing everyone that you’ve actually done so. Even with the mountains of evidence he and later others presented, people still resist the notion that Darwin solved the question of where species came from.

      Liked by 2 people

      • Newton’s gravity is a great example to bring up here. If you try to put yourself in the position of those who were hearing about it for the first time, it would seem fairly strange to say the least.

        On Descartes, I can see why people might attribute zombies to him, but I think we should be careful in this. He thought we ourselves were mostly automata—i.e., mechanistic—with the exception of our capacity to reason. Animals, for him, are automata, utterly unconscious. (Geordie Bear is utterly offended, he assures me in his British accent.) We are automata in the sense that we possess “animal spirits”—reflexes, for instance—that we don’t consciously control. Then there’s the curious matter of the pineal gland, the seat of the soul in the brain. It’s absurd, sure, but if we can step back in time, we see this signifies a step toward the reduction of consciousness (what Descartes calls the soul) to the brain. Actually just one funny portion of the brain. So now imagine if you took Descartes’ argument and stripped it of its God stuff, changed the word ‘soul’ to ‘consciousness’ or ‘psyche’…perhaps we shouldn’t. I get that point. But when I read him, I sense his motivations and interests coming together more concretely when I ignore the circular argument and rehashed rhetoric, and focus on the novel ideas. He was definitely ahead of his time, but it’s so easy (and fun) to make fun of him.

        I don’t know why I’m defending him. I don’t even like him and I blasted him in my undergrad thesis. Maybe it’s just a reflex. 😉

        Well, more on your post in a separate comment…

        Liked by 3 people

        • I didn’t mean to disparage Descartes, although rereading my comment above, I can see where it came across that way. I think for his time, he reasoned much further than anyone else could. He didn’t know about electricity, cells, or computational systems, so for him, plausible physical processes for the mind were nonexistent.

          Given that landscape, who can knock him for concluding that there had to be something to the mind beyond physics? He actually had a good reason for supposing that the pineal gland was the soul antenna. It appeared to be the only thing in the brain that there wasn’t duplicated in both the left and right brain hemispheres. (In reality it also has two hemispheres, but it reportedly takes a microscope to know that.)

          I also don’t think he set out to find an aspirational conception of consciousness. (Quite the opposite from what I understand.) But he had a dearth of information to work with, and when humans are in that position, it’s extremely difficult to avoid falling into the trap of reasoning into your cultural indoctrinations. I’m sure a historian 500 years from now looking at our reasoning will be amused by how much we ourselves do it without even realizing it.

          So no worries on defending Descartes. I understand completely.

          Liked by 2 people

          • Regarding Descartes, I’m certainly a supporter, but am I crediting him with an idea that’s fully his? When he said, “I think, therefore I am,” didn’t he mean that the only thing that he could possibly know to be true, was that his thought itself existed? I consider this apparent truth to have tremendous epistemological implications that often seems overlooked today. I take this to mean that we shouldn’t use terms such as “truth” and “knowledge” in absolute ways (unless referencing “I think,” or perhaps things which are true by definition), since it should be more accurate to talk about what we “believe.” It shouldn’t be so difficult for us to accurately say “I believe X,” rather than inaccurately say “X is true.”

            Liked by 1 person

      • Thanks for the encouragement Mike. And yes, the difficult end of straightening out consciousness might be less about developing useful ideas (which should nevertheless be crucial), and more about being able to demonstrate the value of those ideas to a community which might rather remain as it is. If we add up psychology, psychiatry, sociology, cognitive science, philosophy, and so on, they should have a great deal of natural momentum which isn’t easily altered.

        I’m sure that most such professionals believe that “meticulous scientific investigation… will increasingly sharpen the picture [of consciousness].” But how could things be healthy in these fields today, for example, given that things like p-zombies and panpsychism are given so much attention? Observe that reading through the Wiki consciousness page provides a virtual treasure trove of “We don’t know, we don’t know…” statements. Instead of demanding yet more of our “meticulous science,” I suspect that consciousness will finally need some practical models and useful associated definitions — or exactly what Newton provided physics.

        Liked by 1 person

        • Eric,
          Your question about the health of these fields reminds me how many fields have dichotomies between the scientific part and the speculative factions. We see that in economics, where most of what the public hears about is ideological posturing instead of the actual science, or archaeology where a lot of people with fringe agendas hijack the discussions.

          My point about scientific investigation, is that there is a lot of good scientific work going on discovering how brains process information and produce behavior. But a lot of what gets into the pubic sphere comes from a separate faction that wants to talk about the things you noted. I’d like to say scientific results should eventually shrink the space for that speculation, but based on the current disconnect between them, I’m not sure I can.

          Like

    • Sorry Eric. Not sure why I called you “Alex” in the previous reply. Might have had something to do with just having read an email from someone named Alex.

      Liked by 1 person

      • Mike I smiled at your little name mix up with me. While you shouldn’t yet know very much about me, your site may have already given me a pretty good sense of who you happen to be. Hopefully you’ll get to know me soon enough as well, since friendships generally need to be two way streets. (Furthermore there is also this “consciousness” stuff that you’ve said you hope I straighten out for humanity, as well as mentioned that even good ideas can be hard to sell. Thus I’d love to some day teach you how my ideas work, not only to get your input, but potentially your support!)

        Liked by 1 person

  5. keithnoback says:

    Not conceivable. Next.
    Actually, I think the zombie argument is very instructive.
    At the heart of epiphenomenalism is the notion that it is possible to tease apart the quality of an experience from its distinctiveness.
    As you have pointed out, that turns out to be incredibly difficult.

    Liked by 1 person

    • I think that’s why I found studying animal consciousness to be instructive. We’re too close to our own experience. We have trouble separating ourselves from it, and have this feeling that there must be something more to it than just the information processing. But Descartes reportedly didn’t have that issue with animals. He was perfectly happy to imagine them as automatons. But the idea that he himself was a more sophisticated automaton was inconceivable.

      Liked by 2 people

  6. Now on zombies. Thanks for clarifying the whole zombie thing! I get turned off by what looks like algebra and nonsense combined in one crazy thought experiment. Now I think I have a handle on things.

    I don’t think you’re missing anything, from what you’ve explained here. In fact, I’d say you’ve added something to the discussion in taking an epistemological turn. Why do we keep positing these ridiculous conscious objects without considering how we come to know consciousness? Without saying what consciousness is? And there’s another problem. It’s hard to define consciousness in some sort of neutral way, without presupposing the conclusion.

    Zombies don’t make the old arguments more exciting for me, but maybe they do for others. I have no idea. I find it muddies things up.

    Liked by 2 people

    • Thanks Tina. This post was inspired by a conversation I had on another blog with someone who insisted that it was a meaningful concept, although it was far from the first such conversation I’ve had on the topic.

      Like many philosophical thought experiments on consciousness, it seems to be something of a Rorschach test, confirming some people’s intuitions and being irrelevant to others, not really changing anyone’s mind. To be fair, these thought experiments can be clarifying when considering why they don’t work. (So maybe I was a little harsh with labeling the concept “pointless”.)

      I’m reminded of things like the Chinese Room or Mary’s Room. Both are convincing for anyone who already agrees with their author, but not for anyone who doesn’t, and exploring why each group concludes what they do can be instructive.

      Liked by 3 people

      • These types of thought experiments—I’m not sure what to call them, popular thought experiments?—are also good at bringing interest to topics that might otherwise be seen as too technical or boring. They give you the concept in a clever nugget, and it’s more fun to talk about a zombie than a generic ‘automaton.’ But yeah, once the metaphor has been stretched, once you have to have various letters before your zombie, it’s time to let it go.

        Liked by 2 people

  7. Hi Mike,

    I agree with you to a point but I don’t think you’re being fair to the p-zombie argument.

    Like you, I think the p-zombie argument fails because it is circular. The conclusion is baked into the premises. If, for instance, functionalism is true, then a p-zombie is incoherent, so the p-zombie argument cannot be used to argue that functionalism is false.

    However it is a powerful way of articulating an anti-functionalist point of view, so it has value in that sense.

    You’re also right that the p-zombie argument depends on epiphenomenalism (or rather that the p-zombie thought experiment is a way of articulating an epiphenomenalist view). Where I think you go wrong is where you say that we know epiphenomenalism is false. We don’t.

    Our talking about consciousness and typing about consciousness and so on can be explained without recourse to actual consciousness. It arises because of the firing of neurons. Just as with an unconscious AI which can pass the Turing Test, there is no reason that these behaviours could not be caused by unconscious processes, especially if functionalism is in question. Chalmers’ view as expressed in the p-zombie argument is that we can explain all our talk of consciousness and so on by solving the “easy problems of consciousness”, i.e. those problems amenable to functionalist accounts. The problem that remains is how to show that we are not just behaving as if but that we are actually conscious, and if there is a Hard Problem as Chalmers thinks, then epiphenomenalism must be true.

    I agree with you that epiphenomenalism is false, by the way, but it remains a tenable position in philosophy of mind.

    Liked by 2 people

    • Hi DM,
      I may be too embedded in my functionalism to see how the p-zombie concept argues against it. Can you recommend any resources that present that argument?

      I guess I’m having trouble seeing the conceptual space where we can discuss our conscious experience and still have that experience be an epiphenomenon. The only way I can see it is if conscious experience is wholly separate and apart from the causal framework that leads to behavior, but that its states just happen to correspond, to correlate with the states described by the non-conscious activity of us discussing it, without any interaction between the two.

      While conceivable, it seems to put us well into deism type territory, in the sense of allowing us to retain a concept at the cost of it being forever untestable. Or is there another way to conceive this that I’m missing?

      As an aside, I think my biggest issue with Chalmers’s argument is that he seems unwilling to unpack words like “experience”. I fully understand that experience is subjectively irreducible, but assuming that makes it also objectively irreducible strikes me as a category error. It seems to me that, as soon as we make an effort to do that unpacking, the hard problem breaks down into plausibly manageable chunks. But the subjectively irreducible aspect of it means that many will always insist we haven’t really solved anything.

      Like

      • Hi Mike,

        I may be too embedded in my functionalism to see how the p-zombie concept argues against it.

        I see it more as an evocative illustration of that kind of thinking than an argument against functionalism. It’s an effective intuition pump (as Dennett would say), but pumps in the wrong direction in my view.

        Can you recommend any resources that present that argument?

        Chalmers, I’d say.

        I haven’t fully read this stuff myself but I see where he is coming from.

        There’s this paper which argues that we can say something about what is possible from what is conceivable, and outlines the zombie argument in an appendix.

        http://consc.net/papers/conceivability.html

        Basically, the best version of the argument goes: if we can find no logical contradiction in the idea of a p-zombie (and we can’t, I would say), and if there is in fact no such logical contradiction, then it is logically possible for p-zombies to exist. If it is logically possible for p-zombies to exist, then consciousness cannot be entailed by our physical bodies and must require something else to be realised.

        but that its states just happen to correspond, to correlate with the states described by the non-conscious activity of us discussing it, without any interaction between the two.

        Epiphemomenalism does not hold that there is no interaction, but that the interaction is one-way. Consciousness is like a shadow of someone walking. The walker causes the shadow but the shadow (usually) has no effect on the walker. So the correlation isn’t that much of a problem. It’s not coincidental. The idea is consciousness is a byproduct of the operation of the brain but has no causal role. I’m semi-on-board with that.

        Liked by 2 people

        • Hi DM,

          Thanks for the concise description of Chalmers’ p-zombie thought experiment above — I think I understand. You said:

          Basically, the best version of the argument goes: if we can find no logical contradiction in the idea of a p-zombie (and we can’t, I would say), and if there is in fact no such logical contradiction, then it is logically possible for p-zombies to exist. If it is logically possible for p-zombies to exist, then consciousness cannot be entailed by our physical bodies and must require something else to be realised.

          Yes that does make sense to me. Nevertheless from my own “naturalism” belief, I’d say that I am able to find a logical contradiction to the idea of p-zombies. If reality functions by means of a purely “causal” process, then logic suggests that a p-zombie cannot function exactly as I do without being exactly like me. Exchanging one different molecule (or location, or time) would cause something to be different from me in such ways. A being which actually lacks consciousness however, should logically be quite different from me, given the premise of naturalism. Otherwise I agree.

          Furthermore I also enjoyed your “shadow” analogy for epiphenomenalism. Once again my belief in perfect causality prevents me from going that way. Actually I suspect that consciousness evolved essentially as a punishment/reward engineering dynamic that remains utterly crucial to my own function. Given that modern science has no effective model of “mind,” “non-conscious mind,” and “conscious mind” however, thought experiments such as these do seem perfectly appropriate today.

          Like

          • Hi Eric,

            Thanks,

            Nevertheless from my own “naturalism” belief, I’d say that I am able to find a logical contradiction to the idea of p-zombies.

            Right. So you can take the p-zombie argument to be an argument against naturalism, or at least naturalism as you conceive of it. That doesn’t mean that you have found a logical contradiction in the concept of a p-zombie, unless you can show that naturalism as you conceive of it has to be true.

            Like

          • DM,
            We seem to be on the same page, but using different conceptions for “logical contradiction.” While you seem to be using it as a way to prove an aspect of reality, I was using it as a way to infer various things given specific beliefs. As I mentioned above in a homage to René Descartes, there seems to be only one thing about reality that I can ever know for certain about reality, and “that naturalism is true” ain’t it! Of course I can know things like 2+2=4, but I consider them to be properties of human language rather than reality itself. As I was using “logical contradiction,” this will be the case for a naturalist who also believes that p-zombies are possible — one belief doesn’t seem consistent with the other.

            Like

          • Strict internal logical consistency of the p-zombie concept is the kind that is relevant, not whether they are consistent with other concepts such as naturalism.

            If p-zombies really are logically consistent, then naturalism must be false.

            It’s more or less impossible to show that p-zombies are logically inconsistent, though, until you know what consciousness is. We only have different hypotheses. If functionalism is true, for instance, then consciousness is a set of functional abilities. This would mean that p-zombies are logically inconsistent, because p-zombies are supposed to have all the functional abilities that comprise consciousness without having consciousness.

            But you need to be able to demonstrate that internal logical inconsistency, because if they are not logically inconsistent then either we must be p-zombies ourselves or naturalism has to be false.

            Like

          • DM,
            I don’t want to “strawman you,” regarding your assertion that “Strict internal logical consistency of the p-zombie concept is the kind that is relevant, not whether they are consistent with other concepts such as naturalism.” I suspect that I disagree, but might not sufficiently understand what you mean. Therefore I’ve decided to provide what I call my two principles of epistemology (which I believe would clean up many academic pursuits if formally adopted), and then you can decide if these principles happen to be reasonable.

            The first is that there are no “true” definitions, but only those which are more and less “useful” regarding a given author’s argument. Thus I believe we that we should stop trying to determine what “is” time, space, life, consciousness, and so on. Even the number “two,” as I see it, doesn’t have a true definition, but rather just a very useful and unique one. I suspect that my own definition of the conscious mind happens to be extremely useful, for example, though don’t consider any to be “true.”

            Then my second principle of epistemology is that there is only one process by which anything conscious, consciously figures anything out: It takes what it thinks it knows (evidence), and then uses this to assess what it’s not so sure about (theory). The more that evidence continues to remain consistent with a theory, the more that it tends to become believed. (I’ve been told that there are plenty of other ways to consciously figure things out, though no one has yet offered an example.)

            It is by these two principles that my long earned belief in naturalism, suggests that p-zombies cannot exist.

            Like

          • Hi Eric,
            Just jumping in to this interesting discussion to note that your first principle sounds very similar to instrumentalism. Instrumentalism holds that scientific theories are only pragmatic frameworks for making predictions but do not necessarily reflect reality. That’s as opposed to realism, the view that scientific theories reflect reality to at least some level of approximation.

            Most scientists are realists, I think because they’re primarily motivated in their endeavors to understand reality rather than to build pragmatic predictive frameworks. I have sympathy with the realist view, but I think it’s important to be able to shift mental gears from time to time and put on the instrumentalist hat when evaluating theories or potential theories.

            From what I’ve read, an instrumentalist outlook was crucial to the early development of quantum physics theory. (Of course, it left us with an array of interpretations on whats “really” going on.)

            Liked by 2 people

          • Mike it’s no surprise that our beliefs correspond regarding your interpretation of instrumentalism and realism. Yes in our quests to understand reality, we can only develop pragmatic frameworks rather than “truth.” (I have no problem calling experimentally successful theory “potential approximations of reality” however, if that’s how “realism” is considered.) My first principle of epistemology is a practical rule which lies within the process of developing pragmatic frameworks. It seems to me that people commonly fail to communicate with each other, largely given their use of separate definitions for various associated terms. This simply doesn’t work. Instead I believe that we must take someone’s definitions as “true by definition,” and from that point try to understand his/her ideas.

            Given how clear it is that there are no reality based “true” definitions (as opposed to language based), once I ended my academic hermitude I figured that at least this principle wasn’t going to be a difficult sell. In fact I presumed that there must already be a prominent person who claims it, even if widely ignored in practice. Since I’ve not yet found an original “owner” of the idea, however, then why not advertise this principle (a first of two) as my own?

            I suspect that even modern physicists are in need of it, that is if they’re still looking for “true” definitions of time, space, and so on. I’d have them take the separate path of directly defining such critical terms however they see fit, and then checking to see if what they’ve developed seems useful. Do such models jibe with evidence? My position, for example, is that we must try not to ask “What is force?” but rather “What’s a useful definition for this humanly fabricated notion?” Newton defined it as a product of mass and acceleration, but only “usefully” so, I think, not “truly.” Thus if need be, for our own models we should feel free to define “force” in separate ways.

            Liked by 1 person

          • Eric, I totally agree that definitions are inherently relativist. (Don’t know if you made it all the way back to my post on this.) Too many philosophical or scientific debates amount to people arguing past each other with differing definitions for the common terms they’re using. The typical example is the debate between free will compatibilists and incompatibilists. Most of the participants understand it’s a definitional issue, yet the debate continues.

            Not sure on the issue with physicists, although I do know that there is an ongoing debate in some circles about whether or not time is emergent. I do know if you talk with a typical physicist about words like “force”, they’ll give a generic definition about pulls and pushes to incorporate all the things called forces, such as gravitation, electromagnetism, the strong nuclear force, and the weak nuclear force. The real difficult one seems to be “energy”, which is often described as the ability to do work, although that can get into questions of, available for who, or what?

            Myself, I have issues with the most common definitions of “entropy”, because they seem to imply value judgements about what is “ordered” or “disordered”. In my own mind, I tend to define entropy as how close things are to the end state that the laws of physics want them to eventually be in, but I realize that definition has problems too.

            Liked by 1 person

          • Mike,
            It’s good to see that the nature of definition is also important to you. No I hadn’t previously made it back to that post, but good job! Given that there are no true definitions, I’d have us all stop trying to figure out what consciousness “is,” but rather define it however one likes, and know that the definition cannot possibly be false. Instead it must be judged by its “usefulness.”

            Secondly, I’m as hard a determinist as they come, essentially as Einstein was. (Though I have no problem with Heisenberg’s Uncertainty Principal itself, “God playing dice” would be unnatural.) But to the extent that my perspective happens to be limited, I’m also effectively “free.” It’s strange to me that this is controversial.

            Time emergent? Well yes it should be from a non perfect perspective. But naturalism suggests that all things reduce in the end, not that we idiot humans can reduce them.

            Regarding those four different kinds of “inherent forces” that you’ve mentioned, I presume that they all function as if there’s a mass which accelerates, even when it has no change in velocity. While lying in bed I should still be “effectively accelerating” into the Earth, for example.

            I believe they define “work” as the product of force and distance, so this shouldn’t be subjective. As for the “non useful work” associate with entropy — I just don’t see how that idea could be objective. Useful to what? Like you I also don’t know what they mean by “order.”

            Liked by 1 person

          • Eric,
            On hard determinism, just out of curiosity, do you favor any one interpretation of quantum mechanics?

            Any time I use the word “emergent”, unless I stipulate otherwise, you can assume I mean weak emergence, the epistemological or pragmatic version. It hasn’t been empirically verified that everything reduces to elementary particles and fundamental interactions, but I don’t see how it ever could be, and I’d have to call the theory that it does a strong one.

            On force always being mass and acceleration, good point. It had never occurred to me that was the case. Thanks for pointing it out to me.

            Liked by 1 person

          • Mike,

            I only have a very basic understanding of QM, but it runs through my head at least like this:

            Matter seems best to consider as neither particle nor wave, but rather both. We measure particles and waves in separate ways, and when we do so, the more determined things become in one regard, the more speculative they become in the other. So it would seem that we cannot, even theoretically, measure anything perfectly.

            And now the controversial part:

            From this most physicists today seem to decide, “Ah… the uncertainty observed here must then reflect a fundamental randomness to nature.” I consider this quite an arrogant bit of metaphysical speculation — as if we idiots with our pathetic tools, can determine that nature has a fundamental element of randomness to it. Of course they might be right about that, but I hope they also realize that what they’re theorizing, can quite usefully be considered a void in causality, or “magic.” Einstein was not as arrogant as this.

            Thanks for your clarification on emergence. I happen to be a very literal person, so I do need qualifiers such as that from time to time.

            Then as far as “force” goes, thanks! Here I’ll once again mention that Newton only gave us a “useful” definition of force rather than a “true” one. We can and do define it usefully in other ways as well.

            Liked by 1 person

    • Actually, I’m not sure that I agree that epiphenomalism is false. It kind of depends on what we mean.

      On the one hand, I seem to disagree with epiphenomenalism because I don’t think something could process information in the way we do without being conscious (because I think to be conscious is just to process information something like a human processes information).

      But on the other hand I seem to agree with epiphenomenalism because I don’t think that the fact that we are conscious is needed to explain anything we do — an explanation in terms of neurons firing and so on is sufficient, so consciousness has no explicit causal role. It’s just a label we give to an aggregation of lower level physical (or logical) causes.

      This either agrees with or disagrees with epiphenomenalism, depending on how you look at it. I guess it unasks the question.

      Mu!

      Liked by 1 person

      • Hi DM,
        I think there are two questions here, and we should probably be careful not to conflate them.

        One is whether consciousness controls actions. This gets into exactly what we mean by “consciousness”. If we see it as the totality of information processing the human brain does, then certainly it controls our actions.

        But if we narrow it to awareness, then saying that awareness doesn’t control our actions seems like a very coherent statement. I do think it has causal influence, but the executive centers of the brain seem capable of including or not including awareness depending on the circumstances.

        My own feeling on this, which I’ve only recently started to develop, is that we become explicitly aware of something when we have conflicting mental reflexes about how to respond and have to make trade off decisions between them. When the response can be a simple mental reflex (or must be due to time constraints), what we think of as awareness seems to have less of a role, at least before the action.

        But all of that seems distinct from conscious experience having no causal role whatsoever. I can’t see how that is true. What exactly would be going on when someone talks about the redness of red or the painfulness of pain? It seems like epiphenomenalism argues that the actual redness of red or painfulness of pain doesn’t cause us to say the things we say about them. But then what is causing us to say those things? Neural processes? But if those neural processes don’t include the actual experiences, why would it cause us to say those things? Why would we say anything about them at all?

        Like

        • I think it’s only the latter question we are really talking about. What function does phenomenal awareness serve? Couldn’t we get by without phenomenal awareness, but just the kind of functional awareness computers have?

          What exactly would be going on when someone talks about the redness of red or the painfulness of pain?

          My explanation might not agree with what Chalmers would say, because basically I agree with you and not with him, but something like this:

          The system has functional access to the fact that something appears red and that it is in the visual field. The system is under an “illusion” (if we can say that an unconscious system can be under an illusion) caused by the architecture of its visual and other subsystems that this awareness lets it perceive something beyond these bare facts. In other words, it has a false belief (or a false internal representation, if you don’t think that unconscious systems can have beliefs) that it perceives an ineffable quale called redness, so it reports this. The system does not understand how a functional account can be responsible for what it thinks it perceives.

          Liked by 1 person

          • Thanks DM. I fully understand you’re clarifying an outlook that isn’t yours, so I appreciate it. I’ll admit that we can’t eliminate the possibility of that arrangement, nor the one I sketched above. But I think if we have to resort to bolting such strange conceptions onto the original notion, it only shows how strained that notion is becoming.

            On getting by without phenomenal awareness, my current feeling is that we can’t, and for computers to be able to have the same behavioral repertoire, they will need to have some form of it. The important question is what that experience actually is. My previous series of posts, particularly the last one, explored this question, but the TL;DR is that I think phenomenal experience is the process of building models of the environment, of ourselves, of the relationship between the two, and of our reflexive reactions to the current content of those models.

            Like

        • paultorek says:

          We do know that epiphenomenalism is false, precisely because it implies that the actual redness of red doesn’t cause us to say things about it. This claim, however, would imply that what we are talking about when we talk about the redness of red, is not some ghostly non-natural quale. What we are talking about is the normal cause of our consciousness-speech: some physical or functional property. Semantics follows the causality. I am not arguing that there are no epiphenomenal properties: only that, if there are, they are not what we’re talking about.

          Liked by 2 people

          • I agree. For epiphenomenalism to be true, any talk of conscious experience must be an illusion and the consciousness that is being spoken of must be an illusion, even if it somehow happens to correspond to the epiphenomenal version. All said, Occam’s razor seems to favor that consciousness has causal effects.

            Liked by 1 person

          • Paul and Mike, you guys are reminding me to the line at the top of the “Conscious Entities” site of Peter Hankins:

            “If consciousness is an illusion, then who is it that’s being fooled?”

            Liked by 2 people

  8. Hariod Brawn says:

    Many thanks for yet another clear and concise perspective, Mike. I wonder if I might get a reaction from you on something that seems to nag me, or persist at the back of my head as a kind of unresolved – or perhaps unresolvable – question. That question is whether we’re not barking up the wrong tree in attempts to arrive at some explanation of consciousness as being material or immaterial in nature, or that such an explanation may require and constitute both as their own categories – your cited Substance Dualism.

    I’m in no position to suggest any other tree to go barking up, only perhaps that illumined consciousness is no more or other than as it presents in and as awareness – is that too naïve, not any kind of explanation? If not, why isn’t it? In other words, must we completely abstract any explanation from what is any case self-evident – i.e. it is what it is! What presents seems something other than material or immaterial, it doesn’t occupy space and yet it isn’t nothing. Of course, things in space (brain functions, the senses) play a role, yet when we direct awareness at itself we apprehend more than functions being performed in space. It intuitively seems non-local, not something, but not nothing. Is this purely a problem of the unreliable witness? Is it right to reject in our analyses the very thing we’re trying to explain as itself being an answer? It’s not an answer we can document in words, but why isn’t it an answer?

    What I detect as I wander around reading various analyses on consciousness is a trend of many to want the subject to be purely a Physics (notwithstanding a far lesser trend towards Panpsychism), which of course imparts a sense of objectivity and reasoned rigour (good things in themselves, fair enough), and there’s the implication that in sticking to a Physics, then it also is adhering to some verifiable truth or actuality; what’s more, that this will be respected, not subject to the sneering and condescending accusations of being ‘mysterious’, or quasi-religious, or anti-scientific. So my question next is whether you think we must define illumined consciousness within this pre-existent explanatory dichotomy of material/immaterial, or could this be a failure of being able to conceptualise the problem differently? Indeed, is there a problem, or does consciousness explain itself by its very presence, much as we find that unsatisfactory?

    Liked by 1 person

    • Thanks Hariod. Let me restate what I perceive to be your questions, so if I’m answering the wrong things, you’ll be able to set me straight.

      I think your first question is why we don’t take awareness in and of itself as its own explanation. In other words, why do we feel the need to attempt to reduce it? This is an excellent question, because I think it is a mirror of the question I usually ask when people talk about the hard problem: why do we just take experience as something that must be explained in whole instead of attempting to break it down into more manageable parts?

      My response to the why-reduce question would be to wonder how we might make any progress on consciousness if we’re not willing to do that. I know many people want consciousness, awareness, experience, to be something fundamental. But we only reached the conclusion about other things we regard as fundamental (electromagnetism, gravitation, etc) as being fundamental once every attempt to find any constituent elements has proven fruitless.

      But here we come to a dilemma. Conscious experience appears to be subjectively irreducible. So the question is, how do we respond to that state of affairs? Do we note its subjective irreducibility and stop there? Or do we explore whether it might be objectively irreducible. The problem of course, is that we’ll never be able to verify any theory of its objective reduction from our first person subjective perspective. Our minds evolved to perceive, not to be the perceived, which gives us serious blind spots for introspection.

      This, it seems to me, is the heart of the hard problem of consciousness. I doubt we’ll ever be able to close the subjective / objective gap. We’re stuck with it. I fear all we can do is clarify its existence. But unless we’re prepared to just give up, it seems like we have to explore where we can, and that means in the objective space. Many will always reject this approach, but other than the mystical stuff you alluded to, I’m not sure what alternatives they can present.

      And that leads to what I perceive to be your second question. Does this investigation have to be physical? I’d say not necessarily, but I do think it needs to be evidence based, and the physical seems to be where evidence is obtainable. As I noted a few posts back, if someone can find evidence for a non-physical aspect of the mind, then we absolutely should incorporate that into our theories. I’m just not aware of any scientifically verifiable instances of that kind of evidence.

      All of which is to say, if we want to understand consciousness, awareness, experience, etc, I think we have to be willing to pierce the veil, to dissect it, and to make progress in any direction that we can get traction. Some of those directions may eventually turn out to be dead-ends, but I’m not sure what the alternative would be if we want to be scientific about it.

      Wow, this turned into an epic comment. Hope it addressed your actual questions, but let me know if it didn’t.

      Liked by 1 person

      • Hariod Brawn says:

        Mike, this is fabulous, thankyou very much indeed. I’ve copied it to a separate file (okay?) to mull over further off your blog, and if I may, will respond should my contrarian side feel it must express its pesky self! In the meantime, I’m very grateful for the time and effort you’ve put into this, and I apologise for the rather rambling nature of my own musings above, which you did well to reduce. 🙂

        Liked by 1 person

  9. v says:

    The chatbots that you mentioned might soon make you revise your view about zombies. Sure, they can’t fool anyone right now, but they are improving. Sooner or later they might get really really convincing, especially as we start to record our behavior in more and more details with social networks and personal digital assistants and who knows what future all-seeing technologies and thus give them more data to model us. Have you watched that one episode of Black Mirror yet, or the entire “Real Humans” series?
    If we ever get to that point, we will no doubt have to ask ourselves if such “fake minds” are actually conscious, self-aware, intelligent. Or are they just zombies? Especially considering they were specifically designed “top-down”, starting from desired behavior and having the specific goal of finding a pattern in it and “faking” it, instead of the “bottom up” approach to producing an AI starting from studying and replicating the inner workings and fine details of a brain and then teaching it behavior, logic and intelligence not as a fundamental quality but as a sort-of by-product, much like humans are taught through communication while growing up.

    Liked by 1 person

    • v says:

      Oh I posted those two TV references, but I totally forgot the first place I saw this idea very nicely exposed – Tom Scott’s dConstruct talk https://www.youtube.com/watch?v=_kBlH-DQsEg
      Yes, it’s somewhat off-topic and yes, it’s probably not the first example of that idea in history of mankind :p But I wanted to post it too anyway.

      Like

    • v says:

      Oops… even more to the point is this talk instead: https://www.youtube.com/watch?v=Zf6C-pZ3heY
      Sorry, been a while since I watched them, getting confused.

      Like

    • Thanks for the video links. I’ll check them out when I’m somewhere where I might be able to watch them.

      On the chat-bot, if it could fool you successfully during days and weeks of discussion, consider what it would have to be able to do. It would have to emulate a human mind, along with that human’s theory of mind, thoroughly enough that you couldn’t tell the difference. We might, for instance, be able to imagine a brute-force lookup table large enough to accomplish that, but could we ever eliminate the possibility that the sum total of that table’s entries coupled with the lookup process isn’t effectively an algorithm that implements consciousness?

      The question, or course, is how long do we give it to convince us that it’s really conscious? It’s not hard for me to imagine a chat-bot that fools 30% of humans in a 5 minute test is still a zombie, but the longer that test goes on, the more difficult it becomes. A chat-bot that someone has a successful one month relationship with, including all the types of discussions you and I might have in a month, would be difficult to consign to zombie status.

      Like

      • v says:

        “A chat-bot that someone has a successful one month relationship with, including all the types of discussions you and I might have in a month, would be difficult to consign to zombie status.”
        Even if it’s based on years of data on previous such conversations between you and the person it is trying to fake?

        Like

      • v says:

        And just to tell my own opinion on the matter, I would instantly acknowledge consciousness in any system that we built “bottom up”, but I still hesitate somewhat for “top down” systems that start with focus on behavior. It’s probably a bit hypocritical, considering I will personally never actually understand the basis for that “bottom up” design, the small scale inner workings of either a real brain or our replication of it, let alone how it all combines to give raise to intelligence as a whole. But it’s somewhat more convincing to know that when you get down to the details it works like the real thing.

        For the “top down” approach, I see your point and I mostly agree – ignoring the internal details and looking at it as a black box, if its behavior is sufficiently close to the original that I can not tell the difference, I should probably treat it the same as the original. But this notion that it’s whole design was always centered around faking the behavior rather than understanding what makes it tick does not let me concede completely.

        I’m very curious to see which approach will yield results sooner…

        Like

        • My intuition is that the design process shouldn’t change our conclusion. Consider that human and animal consciousness evolved with no explicit design process, just random mutation and natural selection across thousands of generations. If we managed to build a system that could reproduce all the functionality of consciousness, short of invoking substance dualism, I can’t see how we could have any certitude that it wasn’t really conscious.

          Not sure on which approach will work sooner. I suspect it will be some hybrid approach involving a system that may have aspects of primal consciousness coupled with functionality designed to fake, say, human level social intelligence.

          Like

  10. Pingback: Why about subjective experience implies anything non-physical? | SelfAwarePatterns

  11. @EdGibney says:

    I just discovered your blog (and added it to my feedly reading list) while researching my own post on the problem of philosophical zombies. I like your observation that the zombie arguments are circular, but I had a slightly different take about whether or not we can ever prove epiphenomenalism is definitively false. I believe that such certainty on these things is inherently unknowable in a universe where we can’t see the future. But that just means zombies are like every other imaginable but unseen thing (e.g. gods, Russell’s teapot, flying spaghetti monsters, etc., etc.) — they don’t prove a thing about the actual universe since they are observations with an n of zero.

    http://www.evphil.com/blog/response-to-thought-experiment-93-zombies

    Liked by 2 people

    • Thanks Ed! Appreciate the quote and link.

      Given that we can discuss our experiences, what about epiphenomenalism do you think keeps it a feasible proposition?

      I like the sound of evolutionary philosophy. Too much philosophical discussion takes place without reference to evolution. Asking why we’re conscious without looking at the evolutionary reasons for brains strikes me as removing a large piece of the puzzle and then wondering why we can’t solve it. Subscribed!

      Liked by 1 person

      • @EdGibney says:

        Ah, really great to have you aboard. As I’ve gone through the history of philosophers and thought experiments I have also found that an evolutionary perspective was all too often sorely lacking. (Fair play to the thinkers, most of the history of the subject took place before Darwin. But after Origin in 1859, there’s not much excuse other than inertia and scholarly walls of separation.)

        As for epiphenomenalism, I should make it clear that I really don’t expect it will be proven correct; I fully place my bets on physicalism and monism. However, I’m not quite sure I follow why you think “discussing our experiences” proves epiphenomenalism is false. Something like the Libet experiment can be interpreted as showing the mental thoughts come after the physical upwellings.

        (By the way, I think that interpretation of that particular argument is crap: http://www.evphil.com/blog/libet-and-another-free-will-thought-experiment)

        What I’m really saying is that while any form of dualism or epiphenomenalism hasn’t been shown yet, we can never know what future discoveries await us. We have to acknowledge the radical skepticism that is created by something like Descartes’ evil demon because we just cannot know the future. What if we are in a computer simulation or an evil demon is deluding all our senses? I think it’s silly to live our lives as if such things were occurring, but we have to give up on philosophical certainty because of them. And that’s what keeps epiphenomenalism alive….if very, very remote.

        Liked by 2 people

        • On why I think discussing our experiences disproves epiphenomenalism, consider what has to happen for us to discuss an experience. We have to have the experience. Information about the experience then has to reach the language centers of the brain, which formulates words to describe that experience. Signals from this region then need to go on to the motor control regions, which drive our vocal cords (if we’re talking about it) or our fingers (if we’re writing about it).

          experience->language->motor centers->behavior

          Epiphenomenalism is the idea that our subjective experience is a side effect, that it has no causal influence on behavior. But our ability to discuss it seems to show that it is part of the causal framework.

          Of course, as I think someone earlier in this thread pointed out, the mechanisms that drive our discussion could correlate but not be caused by subjective experience. That would mean that it isn’t the real redness of red or the painfulness of pain that causes us to talk about redness or painfulness, but some kind of stand in process. But it seems to me that this is an unparsimonious move that can be used to question any apparent causal relationship.

          I do totally agree that we always have to be prepared that a future observation may overturn any conclusion, no matter how solid it seems. For me, that’s an asterisk that always exist for anything we currently think we know.

          Looking forward to reading your posts!

          Like

          • @EdGibney says:

            Okay thanks, I think I understand your point about discussing experience now. I wonder, however, if a staunch epiphenomenalist would say that a zombie, a Chinese room, or an AI computer could also talk about information its sensory apparatus had detected and recorded, without actually having that “feeling of what it’s like” to have felt that experience. So since they could “discuss experience” and we could too, but epiphenomenalists think there’s a difference in the consciousness between humans and zombies/rooms/computers, then discussing experience alone wouldn’t disprove their point. At least I think that’s what they might say. I’m not usually in a position of standing up for their point of view. It’s a slippery idea, which I think we can’t fully interrogate because physicalism precludes us from ever knowing “what it’s like” to be any other mind.

            Liked by 2 people

  12. Hi Ed,
    I’m quite happy to meet you as well. We seem to be on similar missions, and begin them from the physicalist premise. It also interests me that you consider philosophy’s troubles with epistemology to be so problematic, since I’ve recently decided to reformulate my own writings to an epistemology focus. I’m a bit concerned that my dodgy WordPress account won’t be recognized by Weebly, but in the end I may just grab a Weebly site anyway.

    In the mean time if you’re curious you can reach me at: thephilosophereric@gmail.com. We seem to take different approaches, yours apparently from a normative “ought,” and mine from a descriptive “is”. There is a clashing here, since I consider “is” to be all there is. Regardless a discussion should test each of our abilities to consider the issues in objective ways.

    Liked by 1 person

    • @EdGibney says:

      Wow, thanks Eric. I don’t know how much of my website you read, but that’s an incredibly insightful takeaway about my approach. I live in England and am about to shut down before bed, but I will take a look at your website tomorrow. In the meantime, I thought I should share with you the link to a philosophy paper I managed to get published in a peer reviewed journal even though I have no formal degrees in the subject. It’s actually a paper that claims to provide a bridge between is and ought using a method based on Hume’s own insight that reason is the slave of the passions. It’s really the most important thing for the rest of my evolutionary philosophy project, which is why I went to the trouble of getting it published. You can find links to it here:

      http://www.evphil.com/academic-paper.html

      I also consider the following blog post to be my most important one about epistemology. For millennia, we’ve relied on Plato’s definition of knowledge as: 1) justified, 2) true, 3) belief. But that was built on an ancient’s view of the universe as an unchanging and eternal thing. Once we discovered evolution in 1859 and the Big Bang was confirmed by background radiation in the 1960’s, our cosmological revolutions should have led to epistemological revolutions as well, but so far they have not. Gettier challenged Plato’s JTB theory of knowledge, but was unable to replace it. I say that in this changing universe, however, there is no such thing that is eternally TRUE. Therefore, knowledge can only ever be: 1) justified, 2) beliefs, that 3) are surviving. See more here:

      http://www.evphil.com/blog/knowledge-cannot-be-justified-true-belief

      I look forward to lots of discussion on these points and on anything particular of yours that you’d like to point me toward as well. Cheers!

      Liked by 2 people

Your thoughts?

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s