Negative SEO kicks in once a number of links to a site exceeds a threshold. The threshold value is unique to each site but usually #NSEO would involve thousands of links or even millions. I would not be concerned about a few links from a directory even if you have no other links.
Does Google view multiple links from directory categories as negative SEO? Should I disavow these directories? ? 
